 For those of you who are listening in here, one of the wonderful things about this universe of cloud natives and open source people and the community development people is that we're all quite willing to share our stories and help with other projects. And, Tari, you have been instrumental in lots of different arenas in OpenStack and I've followed a lot of the OpenStack stuff from the early days and I'm really thrilled that you're here today to share your lessons learned under what I call the big tent. I think you guys coined the word or it was coined about OpenStack. So I'm going to let you take it away and talk for as long as you like and then we'll bring on a panel as well. So Tari, take it away. We're on time for a change. Perfect. Well, thank you for having me. Thanks to Diane and Daniel for the invitation. It's always great to be able to share our experience and mistakes and good things about the OpenStack experience. So my name is Tari Cara as I'm working for the OpenStack Foundation and I've been working on OpenStack since the very beginning of the project. And in a month, we'll be celebrating the 10 years of the OpenStack project. I think it's really a great time to reflect back on what happened and derive lessons from this adventure from our experience really openly developing at a massive scale. So what do I mean by openly developing at a massive scale? By openly developed, I mean an open source project where anyone can participate in a legal playing field where there is no main sponsor with special rights or special access where everything is transparent. And obviously today, there are a few new projects that have followed the same path but most not only Kubernetes obviously, it's also an openly developed project. But back in 2010 when OpenStack started, it wasn't that common for new projects to be set up that way. And it's also massive in terms of in 10 years, OpenStack has grown a lot and I have a few numbers to try to illustrate that. We currently have 762 Git repositories and those are all official OpenStack repositories under the stewardship of the OpenStack technical committee. It's basically all those repositories under a single governance body. We also have more than 8,000 individual contributors to OpenStack so far. And here I count only code contributors to authors of patches that have been accepted in OpenStack. So not just people that participated in the community, one way or another by asking questions or anything. And in terms of activity, we had around 48,000 changes merged over the past year. And those are Garrett changes which are comparable to GitHub pull requests for those who are more familiar with GitHub. As in there, reviewed by a human and tested before they are merged. So these are like comparable metrics. And this place is OpenStack still as one of the most active open source projects in activity today. Finally, in terms of deployments, our OpenStack user survey reports about 10 million CPU cores of computing power running on OpenStack. Those are pouring virtual machines, containers, bare metal machines, storage and networking resources. And that's only what's been reported to us because we also know a lot of deployments that do not report exactly numbers to us. The picture on the slide is the large Hadron Collider at CERN, the European Center for Nuclear Research, which uses an OpenStack cloud with more than 300,000 CPU cores to power its data processing capabilities. So overall, I would say that those 10 years have been a wild ride. We mostly focused on handling that crazy growth. And there was really a lot of yak shaving involved, especially on the project infrastructure side, where we quickly hit the limits of what we started with in 2010. So we started with the launchpad system from Canonical. Clearly, at one point, we overgrew Bazaar, which was the code revision control system. And so we switched to Git. And then from launchpad, we switched to Jenkins for running our tests to Zoolci for handling the load that we ended up running. So there was really a lot of yak shaving involved. And at the same time, now it's really calmer. And we don't grow as fast anymore. OpenStack components are more stable. So it's really a good time to reflect back and extract some hard-learn lessons from that experience and share them. The first lesson is that you should ignore the headers. And it's sad that it happens with open source projects. But with any reasonably successful project, there will always be a group of people that will publicly express dislike for what you're trying to achieve, what you're doing, and trying to bring you down. And usually it's coming from people that are not really part of the project at the beginning, and they feel left out. So rather than trying to join and participate, they prefer to dislike the project publicly. And OpenStack has been predicted dead for the past nine years. There's been an interesting exercise. But we're still very much around. The option is still growing like crazy. So my advice would be to ignore the people that say you're dead and pursue your goals, because time will ultimately prove you're right and the open source is very resilient. So that's really follow your own light and don't listen too much to people that don't like you. The second lesson is that hype goes up and down. Hype is this excitement around your technology as people try to rub their heads around the technology and try to understand it. So that really generates a lot of participation, a lot of press articles, massive conference attendance, and it's generally fueled by curiosity. But at one point, people start to understand the technology and what it can and cannot be used for. And at that point, hype goes down. And it's generally a painful moment because to support all the participation that this hype phase brings, you tend to build a lot of structure in terms of event size, processes, project infrastructure, and scaling that infrastructure down to more reasonable and sustainable levels can be difficult, especially if you are engaged in multiple years ahead. And another aspect to that is that the open source project in itself is not a business model. And companies getting involved with the project actually need a real business model. So a lot of companies, mostly vendors, tend to join the project early on in hope that it will magically make them relevant in the 21st century without having a clear business model in mind. And when they fail to magically extract money from the project, they readjust their investments, they jump to the next thing, or they completely drop from the game. And at that point, when hype goes down and companies pull out resources, a lot of people will leave your community, and you have to be strong and resilient. What you soon realize is that at the end of the day, you're actually OK as long as you have users. Users is really what ultimately matters, because they will be self-sustaining at some point. And so past that depression, if you stick to the project, you realize that boring is good. Turns out once you remove all the hype, infrastructure software is kind of boring. And hype can be very toxic. There used to be a time where OpenStack would sell press articles, clickbait headlines. And people would watch every move, read every word on mailing list posts, and try to extract drama that wasn't really there. And that was very stressful. And now we're in a much center environment with a lot less external pressure. And we also got rid of poisonous people that got attracted by hype and glory at some point. So I would say we're in a better position now in terms of community health than we were maybe four or five years ago. An important follow-up lesson is that at the peak of the hype, it's easy to get into an echo chamber and think that your tech solution is the end to all things, that it will solve all problems forever. I've seen it happen with OpenStack. I'm now seeing it happen with Kubernetes in a way. Turns out you're not the last thing. There will always be another technology that will come next and steal the hype. And you have to prepare for that. And when that something else comes, it's important to realize that they don't replace you. Technology is really additive. People like to think that one technology can be used for everything, that it can replace everything. Learn it and you will never need to learn anything else. Sell it and you will just solve all of your customers' problems forever. But reality is more complex. Containers did not replace virtual machines, which did not replace bare metal machines. Functions will not replace containers. Unicornos will not replace function. Technology is additive. And as we add those technologies, integration becomes the real challenge. And that's why it's so important to engage beyond your community. That's what I'm trying to do here by sharing our experience. At peak hype, there is a tendency to think that hopefully things should be absorbed and done inside a single community and ignore all the other communities. Engaging beyond gives you perspective. It avoids you getting stuck into an echo chamber. It allows you to learn from history, learn from the mistakes of others or their good experiences. It reduces the risk to repeat the same mistakes. And that's why I'm here today, to try to share lessons from the OpenStack experience. Another lesson, important lesson is that being massive does not prevent you from being fragile. When you have thousands of contributors like OpenStack has and Kubernetes has, it's easy to think that your community is stronger than it really is. And fact is, you actually rely on the work of a very limited number of highly engaged contributors that take on strategic contributions and critical functions. What we call horizontal work in OpenStack or what the Kubernetes community calls shopwood carry water. I made this graph a couple of years ago to illustrate that. It's a graph showing the number of developers necessary to produce a given percentage of the changes in the OpenStack release. So on that release, we had about 2,500 contributors, which sounds like a huge number, but obviously there is a long tail of very small contributors. So you can see that it takes about 500 developers to produce 85% of the changes. But more interestingly, 50% of the code is produced by just 93 key contributors. That's actually a lot more brittle than the thousands numbers that I started with. And that thousands number can make you believe that you're invincible. Nothing can happen to your community. But the reality is you will rely on the work of a lot smaller number of contributors. That's because some roles are just hard to feel. There are multiple reasons for that. First, organizations tend to do tactical contributions. They tend to invest in things that benefits them in particular, like adding a feature they need or fixing a bug that would only affect them. And you need people to do strategic contributions, things that will benefit everyone, like handling release management or fixing a complex bug that affects everyone. The second reason is that at the peak of the hype, certain functions are really hard to feel. Developers tend to be kings and they have the luxury to choose. So they generally pick things that are less boring and more creative, which barely leaves a number of holes to be filled. And finally, some functions are just requiring significant expertise that is different from that of the crowd, like technical writing or UI design. And it's sometimes hard to find resources to feel those or to attract them in your community. The next lesson is that over time, people will join. One error we made early on was to assume that our principles and our culture would naturally transmit to newcomers who are oral tradition. And that worked to an extent, but at one point it stopped working and we realized that we should have documented our culture, rather than just assume that it will naturally transfer to people that join. The mirror lesson from people will join is that over time people will leave. And life happens. I mean, you can't expect your small circle of leaders to stay forever. And at that point, you need to have a bench of other people ready to step up. You need to train them to understand that they will be stewards rather than kings, that they will have duties rather than rights. And you need to provide multiple opportunities to step up and give proper recognition to those who do. And I think that's an area where the communities did a really great job with the shadows system that allow you to grow leaders in the shadow of existing leaders. And I think it's a really great model to emulate. Another issue we encountered is how we scale development itself. In OpenStack, we split development across project teams that are dedicated to specific components. And in those, there would be core reviewers which would be basically people that who approve the changes ultimately. But those people can't review everything. But they basically have to trust each other to review changes like they would have reviewed them. And turns out that works until the group reaches a certain size, which is around 16 people. You can't really scale up trust. You have to scale it out by further splitting code areas. And that can really be difficult when people don't want to let go of their ownership of a specific area. And that's just one way Conway's Law will bite you. Conway's Law states that organizations which design systems are constrained to produce designs which are copies of the communication structures of those organizations. Basically, what you produce is completely predetermined by how you organize things internally. And in OpenStack, we organized development around project teams, specialized in specific components which made it hard to do cross-project horizontal work. It's harder to track that kind of work. It's harder to do, and it's not well recognized. So as projects got more stable, it started biting us because we wanted the various components to work better together. And so we needed a lot of that consistency work to be done. And that work is much more difficult than it should be. It's important to be aware of that. Another way Conway's Law will bite you is that due to the way the open source project is organized, everything our community developed was put in the same bucket and called OpenStack. And now we realize that it's sending a pretty confusing message. People are still asking us, what is OpenStack today? And it also made it difficult to discover components that can be used standalone without any other OpenStack components. So we've started to recognizing that and make it easier to discover and reuse individual components within the OpenStack framework. We also started to work on other open infrastructure technologies and try not to call everything OpenStack because at one point it's more confusing than helping. So having seen a few of those openly developed, massive openly developed projects now, there are a few axioms that seem to apply every time and I thought it would be useful to try to list them. So the first axiom is that you have to have four opens for it to be really an open collaboration. The first open is open source, which is an obvious first one. Using an open source initiative approved license and not supporting open core models is pretty important if you want to build a healthy community. Open development is the second one. Everything happening in development should be transparent and accessible to everyone. But that's pretty standard in 2020, especially with GitHub that helps setting up projects in a pretty open way. Open design is a step beyond that. Design should not be done behind closed doors by a separate group of privileged developers. It should be done in the open and be inclusive of users and that's pretty critical. And finally, the last open is open community. Any individual should be able to join the community and become a leader. Contribution should be your only currency. Nobody should have special rights based on who they happen to work for. Those four opens are the key to a successful and healthy open collaboration. Open collaborations have a hidden secret which is that openly collaborating has a cost. You will go slower and a focused startup will get to something working in a fraction of the time it will take for an open collaboration to get there. So you have to accept that. You have to accept to go slower because you will also go farther, which is for software, what ultimately matters. Another, sorry. Another hidden secret of open collaboration is that open collaboration will inevitably lead to scope creep. The reason is it's really difficult to say no in an open collaboration. Someone comes up with a feature to support a corner case and brings it to the comments. It's really difficult to reject it. So you end up usually with lots of options, lots of plugins, lots of things that are beyond your scope. You should try to put barriers in place to resist that like an early manifesto that will say what your project is and what your project is not that you can easily refer to afterwards because once the open collaboration is in place, it's really difficult to resist. As engineers, we like to gather data and measure performance to try to improve. And as humans, we like to see how we compare to each other. So that's usually why we derive stats and metrics to follow our development efforts. And we put up websites which present those metrics. The problem is it usually incentivizes gaming the stats and results in the wrong behaviors. The good art flow states that when a measure becomes a target, it ceases to be a good measure. So basically, if you start publishing the metrics and people start to compare each other using those metrics, then the metrics go bad. And it's a difficult one to fix. In OpenStack, we try to avoid it completely by not providing such competitive metrics at all, but then someone else built the website that allowed people to compare to each other. That's really a problem. It resulted in people submitting patches for typos everywhere to try to boost their standing. And I don't have a great solution for that. The last axiom is that face-to-face time is necessary. And I know we're running a virtual event here. And as much as I personally like us to be truly globally distributed, some things require spending some time together in person, like building trust, building shared understandings, solving hard problems, agreeing on priorities. The OpenStack Foundation run our first virtual developer gatherings two weeks ago. And it sort of worked as a sink point for the various teams, but it did not generate any long-term alignment or shared understanding. So something was missing. I think it's great that we can have those virtual events. It allows people that cannot travel usually to your events to participate. It's good to have them. But at the same time, if you only rely on virtual events, you will lack the necessary trust-building to make a few fully online communities completely healthy and sustainable. Okay, so last part of this talk is about giving a set of best practices that should be able to be applied to any openly developed open-source project. If you want to start with a good base, my first advice would be to start by documenting a clear set of principles. Open-source projects are ultimately social beasts. People are joining a community, not necessarily a technology. So setting a number of principles in stone from day zero will really help set expectations and build a common culture. In terms of project governance, I have two rules to follow that I advise people to follow is to add governance structures only at levels where final decisions are needed. There shouldn't be any vanity governance bodies because that's not very helpful and that creates a lot of shorn in the community. But you should add enough structures to strongly align with your constituencies. If you have completely separate groups, they should be completely represented by different governance bodies. A lot of projects just start with the BDFL model. But to me, that's an anti-feature. The BDFL model is basically a cargo-cultured non-governance. So sooner or later, you will need proper governance. And the more you wait, the more painful the change would be. Diversity is critical and we already had a few talks about it today. It's critical to build a healthy community. You have enough different perspectives in a community. And it starts with inclusive community processes and tooling. As an example, when OpenStack started, it was mostly between US and Europe-based developers. So we relied heavily on weekly RSE meetings in a European and America-friendly time zone, which made it fairly difficult to contribute efficiently from China or India. And so my advice would be to build for the fact that Earth is not flat. And for example, reduce dependency on synchronous meetings as much as possible. And more generally, you should make sure that your processes and tuning are not actively excluding minority groups. Because if those tunings are put in place by the majority group, obviously they can be very excluding. I said earlier that as long as you have users, you're good. But users don't appear from nowhere. You have to cultivate them if you want them to be part of your community. Getting users engaged and actively participating in the feedback loop is essential. But it's important also to not build developer versus user mentality. One mistake we made early on in OpenStack is to have separate governance representation for developers and users. I feel like it prevented users from being more naturally involved with the developers and participating in that feedback loop. It can be pretty difficult to unwind as time goes on. Code reviews are now pretty common. But you should also adopt a test-driven code-getting, basically prevent code from merging if changes do not pass tests. So from there's zero in OpenStack, we had that saying that if it's not tested, it's broken. And the truth is in any complex system, if untested things are not broken already, they will soon be broken. So I think it's important to assume that if you don't test things, they can very well be broken. Adopting code-getting, like we did in OpenStack, has a lot of side benefits, like the development branch is always usable and it promotes a test-centric culture that is really good. I would also strongly encourage adopting time-based releases over feature-based releases. I feel like it's becoming pretty obvious now that it helps, because in a truly openly developed project, you will really have a hard time predicting what and when features will land. So rather than doing a bad job at feature-based releasing, you should embrace the benefits of time-based releases, which is to have a predictable cycle, clear plan downtime for your community, happy downstream users knowing when their work starts, and events that can be well synchronized with your development cycle. Embrace time-based releases, rather than do a bad job at feature-based releases. And finally, my last advice would be to automate everything you can automate. This has plenty of benefits. Automation will obviously save you a lot of time, avoid a lot of human error, and ensure repeatable processes. But beyond that, automatable tasks are usually boring. And so for your community, people will prefer to work on automating those tasks, rather than do them over and over again. So in OpenStack, we ended up automating everything, even things that don't really make sense automating, like our IRC meeting calendar, or our world-wideest process is completely automated. But it really pays off in the end in terms of making sure that all those boring tasks are covered. And that's it. That concludes this talk. So thanks for watching this. And I think we can open to the panel for questions. Absolutely. Terry, thank you very much for this amazing talk. I think everybody's been chatting amongst themselves here in there and just going, there are so many things that we all learned and are still learning from the OpenStack Foundation and the work that you guys have done. And I really can't thank you enough for making this actually happen. So here I'm just going to unmute a bunch of people who I think you all recognize and know. It should have everybody unmuted. Rain and Amy Merrick and Lisa Marie and Daniela Squaredo and myself. And I can see sort of the top of your head in that picture, but in your eyeballs. There you go. It went to my other webcam for some reason. There you go. Hello. And it's the rule of the hoodies here. So I'm liking this day today. So we're getting a whole bunch of hoodies going on here. Terry, you said a ton of things that were incredibly useful and lessons that I think a lot of them, since most of us are also working and collaborating with the CNCF and the Kubernetes community, we also have a lot of us came from the OpenStack community into the Kubernetes and there's still a lot of collaboration and work that goes on between. But we really are grateful for you sharing this experience, your experience and continuing to work and collaborate with everybody. So thank you. That said, I have Daniel, Rain and Lisa Marie and Amy here. I'm sure they all have comments on this. But I'm going to just say right off the bat, the golden rule is that you put down for me is if it's not tested, it's broken. So I think that should be tattooed on every open source project thing. And thank you for that one. Well, it's actually Monty. Of course it's Monty. Yeah, it's not me. Sorry. Can we get through one topic out of Monty? Shout out. I guess we can't. OpenStack still has to have at least one Monty Taylor shout out. Hi, monster. We miss you. So I think one of the things that is also really interesting to me about the OpenStack world is how you are now reinventing yourselves with open infrastructure. And one of the lessons that you learned was like pulling everything under the big tent or the big skirt and labeling it OpenStack. And it's nice to see that the CNCF hasn't done the same thing completely. But can you talk a little bit about the reinvention and I think with Starlinks and some of the other projects that you're pulling in and how that's informing where you're going next. Yeah, so it started by realizing that trying to market a number of things. I mean, how we ended up there is because we had OpenStack, the OpenStack community. And the reason behind the big tent approach was everyone that is part of our community is participating in OpenStack. It was the idea that we should be a very inclusive community and who are we to decide who is part of our community and who is not part of our community. So anything that the OpenStack community produced ended up being called OpenStack. And obviously that's not that pretty poor product management because in the end you end up with something that is called OpenStack but has so many facets. And at one point, I mean, I understand why we ended up this way but at one point it started to be more destructive when we started to explain what OpenStack is in clear terms to have more of a brand rather than a product. And so we realized that especially once some of the components that we produced were equally applying to Kubernetes and OpenStack, for example, didn't make any sense to call that OpenStack. So Airship, for example, is a deployment project and they can deploy OpenStack but they can also deploy other infrastructure stacks. So making it a part of OpenStack makes it extremely difficult to explain. It's easier to explain that it's a deployment system that can deploy OpenStack. In the same way we had Zool, the ZoolCI, which was mentioned in the talk, is also something that was born out of the OpenStack infrastructure but generally useful to others. And so if we just call it a part of OpenStack infrastructure and don't give it its own brand or identity, it's really difficult to explain to newcomers that might be interested in Zool without being interested in OpenStack. So at that point we realized that what really matters is not necessarily the technology that you produce but the people that you assembled to work on those problems. And OpenStack as a community is more a community of people that integrate various pieces of infrastructure, open source infrastructure software. And they will combine OpenStack but also SAF or Kubernetes or others to provide infrastructure for others to build their applications on. And so what happens in OpenStack summits for example was more a gathering of operators of open infrastructure solutions. And as they were discussing new projects, new pieces of software, it made really apparent that the real gain for us was to provide this environment where people can discuss open infrastructure in general rather than just OpenStack. And work on the complex problems which are integration and operations rather than just discuss like development of OpenStack. And that's where the flip switched in my head. Like we are not about producing OpenStack. We are about helping people operate open infrastructure. And so we can do that by helping obviously producing OpenStack but also producing other pieces of software discussing integration with other communities and not just the pieces of software that we produce and that's where the extension came. We're a community of humans solving a problem space not just producing a certain piece of software or a certain brand. And if I can jump in here for a second. From a community standpoint, I remember when this discussion was starting to happen and I feel like as community managers we really pushed the envelope on that one because the conversation sometimes starts in the community before it actually gets to the foundations, right? And it should be that way. But I really have to applaud the OpenStack foundation because we changed the name of our community. We used to be just OpenStack. We changed it to Open Infrastructure and then it became Cloud Native Open Infrastructure. And we started doing that I think years before the actual summit changed its name because that's the conversation that the community wanted to have. And we still did OpenStack stuff. We ran the first ever Zool Meetup and we ran the first ever Cota Containers Meetup and we still really tried to push the technology when it was required. But it became more talking about solutions, the community was users, was contributors, was ops, was architects. It was a whole bunch of different people who were coming and they wanted to talk about complete solutions and really solving a lot of business problems. So we had already done that and by we I mean the San Francisco Bay Area chapter. And I really have to applaud the foundation for taking the feedback and for listening and for then changing the name of the summit to really help us kind of mirror the community. Yeah, I totally agree that it's based on the feedback of the community that we ended up seeing that. At least from the development side of the community, clearly that's around the time where we were in that echo chamber that I described in the talk where you only hear about OpenStack, everything, your world is all OpenStack and you're blind to what happens elsewhere. And so it was really useful, the community really helped bringing that external perspective, especially operators in general, having to operate those stacks of software, they didn't care about who produces it. What they want is the resulting stack to work. And so rather than discuss only one piece of technology in a vacuum, it's much more useful to discuss integration of the various pieces and solving real problems rather than be a bit territorial about what you ended up accepting to discuss or not. Yeah, that's like the Ops Meetup Team. They put on twice yearly events and they're mainly focused around OpenStack and operators, but they're including SEP and they're opening it up to other things in order to meet the needs of the operators and again that complete solution. So okay, we need a bigger, more robust storage. SEP fills the need and a lot of people are using it, so why not include it in the operator meetups? Yeah, I always thought of OpenStack as an umbrella project anyway that included a bunch of specific projects that came together to make something work. Switching, it made complete sense. I was involved with Lisa Marie's group before Open Infrastructure changed its name as well and I was like, that does make sense. And so when OpenStack Foundation changed Open Infrastructure, I was like, yeah, duh. It just made complete sense. I'm wondering if the OpenStack Foundation, like the foundation part of it, is going to change to something like Open Infrastructure Foundation or something else in the future or if that's in the pipeline? Well, that's definitely something that's being discussed. I would say that in order to justify that, we would need to... I would say we don't have enough technology or new technology or new members to actually justify that at the moment. OpenStack is still the main product, but if we, as people, like you said, understand what we're trying to achieve when we talk about Open Infrastructure, there will be more potential for other organizations to join with other pieces of technology that are not connected to OpenStack in particular but are still used to provide infrastructure using Open Source solutions and at that point, I think it would make more sense, but I want to rename just for the sake of renaming. I think it's good that we renamed the summits. I think it's good that we renamed the events, but I would say for renaming the foundation, I think it's important to... It would be a new chapter and to justify the new chapter, it would need significant change. I don't want to change it and not add anything. Yeah, that would be my guess. And from the community standpoint, I got to say people do not care what it's called. I mean, honestly, you don't call it anything you want. Yeah, the name of the foundation is not really important. If you look at the PTG, it was going to be the OpenDev plus PTG. Now, that's been separated into two different events, but the OpenDev part is scale and edge and so many different topics than what would be covered during the PTG, which was very OpenStack specific. So inviting other groups to come and talk has definitely been where the OSF is going. Anything that is infrastructure, if it touches on OpenStack, great, but if it's just a neighboring community, OSF has been very open to inviting them to events. And I'm actually looking really excited and lined up for two of the three so far and just hearing what other people have to say and how those other communities interact with us. Yeah, we've been trying to invite as many like adjacent communities to our events, especially the project team gathering because it allows developers to, like they had their own meeting, but they can mingle with developers from the other communities and that kind of posthmosis between the groups is really interesting. It's difficult to do without being accused of stealing other communities, but I feel like developers don't care. I mean, Ryan has been to a number of PTGs and we've seen the SEF, we've seen other like Gata containers, but also Kingston Fabric and other groups joining because there was significant overlap in their interests and just having them around was really useful to bridge the gap between the communities and discuss potential integration points. And in the end, the software works better if we exchange more. And so I feel like it's the right approach to invite everyone and if we have to take some heat about it. Yeah, definitely easier to do this, but I'm really looking forward to the open dev because we will at least get the mix there or we didn't know the virtual PTG. And so yes, speaking of... Go ahead, Mary. I was just going to bring up something he mentioned earlier. In your presentation where you were talking about that communication is your... And that everybody is part of the community if you're contributing. So I want to point out that, Amy, I spent a lot of time talking about this. When you say contributing, you're not just talking about writing code. There are lots of ways to contribute to a community and people sometimes forget that. And there's so many important roles to play and a lot of people get really intimidated about joining and staying in communities if they're not actually programmers working on a project. I mean, Doc was pointed out earlier. The community will let you know where the gaps in your project are. Doc is often a gap, but there's so many other... What was the name of the group, Amy? The AUG or the ATC? I can't even remember the acronym, but in OpenStack we went and did create a user community, a group of active contributors that aren't actually code. And it was called ZonCode contributors. There's bias in that statement right there, and we had to call it something else. And so the active user community goes well beyond people who are just developers. So I want to make that point so that everybody understands they have a world-playing community, an important one. That's the thing. We keep circling back around to that end-user community and the importance of that in any of our communities or ecosystems. And recognizing their contribution. So I think that's a real key point, Lisa-Marie, thanks. Yeah, and actually the dark side of Githops is that once you turn everything into Git repository, then everything is measured against changes that you produce. And if your work does not translate into a change, then you don't exist. And that's really the dark side of automating everything like we did in OpenStack or driving everything to forget in the others, is that if you don't get out of your way to also count all those other contributions that are really key to having a full perspective on what's happening, it's difficult. And you mentioned the AUC criteria, which is basically having a number of rules to automatically recognize contributors that have participated in those non-measurable ways. So if you moderate the session at the NOPS meetup, you shouldn't even have to ask for to be added to the group. It should be given to you because we all know that if humans have to ask for things, they just don't necessarily do it. So having a way to list all those forms of contributions that should automatically be recognized, even if you don't have a script that will automatically compute the list for you is, I think, a good step that we put in place in OpenStack. It's really easy for a developer-centric community to just ignore that. And I'm talking from the... I'm an offender of that in the early years, just in the light letter. Actually, it's interesting that this... I mentioned it earlier today. I'm working on a COVID-19 initiative with a bunch of data scientists and data scouts and people finding open data sources to use for mapping COVID up here in Canada. And what... It's kind of the square peg in the round-hole thing because they have a website, and the website uses GitLab, so they have the Git-centric stuff, but none of the data scientists or the data people are looking at... They're all clinicians and data scientists, and we're trying to force feed them GitLab and Git processes, and we're going to do it. We're not stopping. We are going to use it. So this is just the nature of the beast that we live in, but it's been interesting to listen today and hear how maybe not all of our automation and our processes work for all things, and there might be some things that we need to go. So it's all good. Well, it's like in OpenStack, and we offered it to other parts of the community under OSCEP, is First Contact SIG. A group of people were going to help you get started in the community. OpenStack Upstream Institute. Live hand-holding and help for two days right before summit or at some of the OpenStack days or now infrastructure days. So the community providing people who are going to help you is so important, because for someone who doesn't know your processes like we use Garrett, I love Garrett over PRs, but most people don't know Garrett or how to install it or anything else. So having people there who are willing to take their time to help you contribute your first patch is awesome, and more and more groups are doing. Amy, I have to give a shout out. Amy has done a phenomenal job building a mentoring program. This is another valuable part of a community. The mentoring program in OpenStack and beyond that she's worked on for years. And these are thankless jobs, and as Tilke was saying, you don't always get the credit. So as community organizers out there, it's super important to really think about these things and then to figure out how you can recognize those members of the community. Another thing I would say quite often the community developers, managers, and the people organizing all these things aren't recognized unless they're doing some sort of documentation or something like that. So it's like you might go and work on a project like OpenStack or one of the many other ones and never make an engineering contribution so you never do that. So it's also interesting. I have some qualms about the gamifying, the badging. We use it a lot in Fedora, Fedora badges and stuff like that. So I have never been a huge fan of it. Do you guys feel like that actually works? Is that something that people are incentivized by? I always feel like it's gamifying thing, just personal opinion. So I love gaming, and so I love that, but I completely see your point of view to kind of, there's a certain rejection of gamification unless you are a gamer type, or unless you fit in those holes to earn those badges. And I get that. I think the only badges I have in Fedora are time based from having made my account and then still being a member and then I happen to have cake with the last F cake, and so I have a cake badge. And otherwise it's not real. Everything else I've done for Fedora is around advocacy work and diversity work that doesn't have a corresponding badge or gamification or award, and it's not that I wouldn't have done that stuff. It's just that it's very difficult to recognize advocacy specifically. I think that's something we should change. Let's do something around that right there. Even in the OUI sessions, people have gotten their companies to send them places early and they just sit in the room while we started saying, whoever answers this question gets a Tim Tam. People started answering questions. Here's a sticker. So gamification can get participation, and in those instances it's really good, but when it's for a company or someone to just say, we've got the most commits, we've got the most that, then it's not a realistic application of a gaming because you're gaming numbers to your benefit, not gaming to help the community if that makes sense. Yeah, I also think, I'm not so much worried about, I mean, we've all seen people gaming systems and stack analytics and other, you know, everything. And I'm the first to say that I love metrics, I love dashboards, I love the insights they give me into what's going on in the community and who's doing what, but they're not the full picture. And I think my point is more about that the things that we can measure, we do, but the things that are not quantifiable, we kind of leave off the map. And one of them is like dev rel or dev advocacy, and often what I see is a little bit of, then those folks tend to have, or us tend to have imposter syndrome. Like we may do a career for 10, 15 years in this, but we never made an engineering contribution so when we go to whatever, we don't have that, you know, the badge of honor or something like that. So I'm not so much worried about people gaming the system, people know that happens, right? People have been in enough of these things. So I think that's one of the things that I am always cognizant of is that all of the other participants in the community, how do they get, how do we recognize them? And, you know, we're able to, you know, let people flag themselves as end users using a production and giving them the podium, sharing that. But it's really, I think, one of the more difficult tasks in community building is making everybody feel like that their contribution counts and get that, you know, especially because everybody, not everybody, a lot of people work for companies that value the contribution label. So I think that's something that's, we still haven't quite figured out yet. I mean it is hard to recognize people and like Tom Feifeld actually did have a script that would go through on the AUCs to some degree. But then it was also reaching out and every time it was reached out to me, for me, the women open stack and later the diversity and inclusion working group, I always provided names of people who came, showed up, and did something, not just that flow in a meeting. You know, you came and helped out at something, you deserve AUC. You know, you helped run the mentoring program at AUC. So it's got to be a two-way street where the community is asking the people who know and the working groups in the CIGS who may not be showing up, even the technical contributors. There's an extra ATC status here, you'll know the correct name for it. But like before DOCS was considered a technical contribution, this was a way of the DOCS contributors getting recognized. So there are ways of doing it, but the communities also have to be open to it and be able to supply the information. And it's not just about being recognized because one of the things that the Open Stack Foundation did, and we brought this all the way up to the board, but they listened, was they started giving this passes to the conferences and scholarships. Because the problem is a lot of times if you're managing a community or you're doing that on your nights and weekends, somebody has to know how to slide about that. A lot of us are not doing, like when I was working at HP, I was never paid to run the Open Stack community and I was running the world's largest Open Stack community twice a month in the evenings. It's all personal time. And I'd written two books on Open Stack and I still wasn't considered a contributor. The Foundation listened and they started giving us passes to the conference. And in some cases, because I was never getting paid to be sent there, they were doing some airfare and things like that. And I really thank them for that. And we did get our little AUC on the badge, our little sticker, and it really meant a lot to be walking around the conference with that imposter syndrome thing hanging over your head to actually be, I'm legitimate. I have a legitimate reason to be here in this community. It meant a lot. It's something that foundations can do. I think the key is to judge engagement rather than, you know, the end result. Like if someone engages and participates in a community, they should be considered a contributor and also apply very liberally the role, not try to restrict it to an elite group because otherwise that way lies madness. And so, like, we use the extra ATC for all the translators, for example. If you submit a translation, it should count as a contribution. And so that's one of the mechanisms we put in place to, that wasn't a Git contribution, but it definitely required a human typing names in a list. But it requires some work to properly track contribution in a community. So sometimes it's so much easier to just, like, rely on Git logs and, you know, don't ask questions. Yeah, I know that Daniel and I, who have been doing some work on, he mentioned it earlier today, it's one thing to collect the metrics, but if you don't have a domain expert or someone who knows the community to apply the metadata or to the understanding of what they're doing in the community, it's a two-part measuring and collection of data and information. So that's always pretty significant. Well, we are a little over our time and our next guest speaker has come. And so I really, again, I'm going to keep saying this. All of you were on this call. Amy, I'm so glad you're with us at Red Hat because I am so going to take you on as a commons person to help us build out a lot of our end user and other parts of our community. So it's wonderful to have you here. Rain, you just departed Red Hat, so good luck at Packet. Thank you so much. And Lisa Marie, I can't tell you how much I respect the work that you've done in the OpenStack community. So thank you again for all of that. And I'm definitely going to have all of you back on for other variations on this theme soon. So thanks again for your work and your efforts. And Terri, I'm going to figure out your lighting situation because you look the healthiest of all of us. I know, right? That color lighting. Look at you. I'm so jealous. No. Totally jealous here. So anyway. Thank you for everything you do. It's not easy what you've done. You have created an incredible community with OpenShift, Gathering, and the conferences that you put on. And I've been honored to participate in, I think, five or six of them. They are really amazing. So thank you for all the work you do. We'll keep on truckin' when we'll be doing more of it and reinventing it.