 Real quick before we get started welcome everyone if you would like to scan this QR code this leads to a survey that's being conducted by project chaos a Linux Foundation project that's working with community metrics The survey author is actually dr. Dawn Foster in the back of the room So love to point her out in the bear sir But yeah, it's really important survey and we're we're encouraging everybody to kind of get in there and provide your thoughts So we appreciate it All right. Well, thank you so much for coming to our session on measuring the impact of community events I'll start by introducing myself. My name is Natalie Paz. Minio. I'm an events coordinator at red hat So I sit under the global events team And just a little context the way that our events team is split up is between proprietary red hat hosted events and industry events And so I sit under the industry event team The community event team working very closely with the open source programs office Right and so my name is Brian profit. I am with the open source program office at red hat Specifically, I am the what I do. What do I do? Oh, yeah, the senior manager of the community outreach team and community outreach Does a lot of different things we do a lot of marketing content production working with events community metrics And and a lot and training and enablement Creation so we do a lot of different Activities that really kind of resonate and resolve into the topic of discussion that we're having today So what we're going to go over so specifically this entire talk is going to be talking about the challenges of how do you measure? Community events for any kind of real impact. How do you measure the impact that your company or your? Project is doing this is going to be addressed for both commercial organizations and also for You know upstream projects, but when you go to an event, how do you know you're getting the most out of the event for your? organization And then we're going to talk about how We are setting up some things at red hat to sort of meet those challenges and there's no real direct solution You kind of have to come at it sideways and we're going to be talking about that And then we're also going to be referring, you know Once we go through the solutions we've come up with we're going to say okay How do you take that and then maximize those benefits and apply them to improve? You know your community and again you're going to hear like me at least bounce back between company organization and project I Should be saying organization Consistently, but if I go back and forth just know that these principles will apply to a commercial run Or stewarded project and also one like coming out of something like the Apache Foundation or the eclipse foundation So just so you know it does go back and forth. So we're going to start off with talking about the challenges of measuring Around community events and what that means All right, and I'll start us off with our first few challenges that we see with the community events The first one being that oftentimes with community events It can feel like there's no direct measurement or less direct measurement when we compare To industry events that we are at and when I say industry event just to define that Those are our typical events where we are showing up to promote our products or our services And so those are generally there to generate sales leads. It's pretty easy to Kind of track exactly what we're getting from that event Tag it to revenue dollars versus a community event looks a lot different and so Unlike an industry event. Nobody is buying anything This is not true a hundred percent of the time There are companies and organizations that come to community events to promote their services But it feels a little bit different because a lot of times it's community first and so maybe we should say they're buying less But because of this change the business can feel less prioritized We're all here to represent our companies and our organizations, but We're we're also here to Network with a group of peers with other from other organizations from friends and we are here for Community open source around shared interest and so that can make it feel like the business is a little bit less prioritized compared to other events and Finally the return on investment can be difficult to determine or identify Because we don't have those sales leads those direct numbers It can make it a little bit challenging to say how are we showing? What is the value of a community event? We can feel it. We know it, but how do we measure that? Next up community events involve multiple organizations And so each company each organization they are different each project is different There's a lot of diversity in that Specifically also in their goals how they show up to events They're going to have different needs and that's going to show out in their event strategy and their event approach some of the goals that We've identified our sales and like I said while this may not be first and foremost at a lot of community events There will be some organizations that are coming here To make sales or just to show folks what products and services that they offer Next is one that's very prominent and very obvious is community building and this could be to build a user base This could be to gain contributor contributors But community building is a very popular and common-seeing goal at a community event Next up is networking organizations might come to network build partnerships To learn more from other organizations to see how we can collaborate together To see how our services and products can complement each other rather than just compete, but networking is Definitely a goal not only on an individual level, but on an organizational level at community events We also have employee seeking opportunities job Recruiting if organizations have a lot of openings available or very specific ones that they're targeting we can see them coming Trying to engage with attendees and interest people into coming to their organization And finally brand awareness and thought leadership. This is another big one. We see in community events Organizations coming to show their support show their commitment to open source and to the community projects And giving back in that way So also also coming to these Events are going to be the actual community projects themselves. So you have not necessarily commercial organizations but the actual project and I may have copied Natalie's slide a little bit if you notice any similarities here and that's actually intentional because a lot of the goals that commercial organizations have Community projects have with slightly different flavors. So instead of sales for instance and commercial opportunities Some community projects are going to be coming to these these events looking for fundraising or at least Putting out the message that they are seeking Some kind of support of our, you know, we've had discussions already today at red hat about providing infrastructure support to or Projects that have nothing to do with red hat more excited for that opportunity So and it could be just general fundraising and then there's also like commercial Goals there's community building in this case community building is more about Taking the community you already have and strengthening it So it is very common for all of us if those of us who are involved in open source projects When we come it to an event that we have some kind of social hour Monday went out with it. I went out with the chaos people Tonight I'm going out with some of the Apache people, you know, this is actual this sounds like oh, we're having fun Yeah, and we are but it's also a genuine part of community building because of the pandemic taught us nothing else is That human connections matter and face-to-face is You know, they're you know virtual conversations are great and I'm glad we had the opportunity to do them You know in the last few years But face-to-face meetings and face-to-face socialization is still a key part of the human experience And that is a big part of what any community needs Networking again, that's kind of a little bit different from community building, but yeah, we're talking to other projects We're seeing what are you doing differently than what we're doing are there ways? We can collaborate with each other the thing that I love the most about the open source world It's that it is not automatically defined that somebody at a company. That is my competitor Is actually going to act like a competitor? I have no problems going over to the Suza booth or the AWS booth or whomever and Saying hey, what are you all working on or I heard you're working on this? We've got something similar. Can we kind of collaborate on this in the upstream and That is actually a real possibility And and again, that's something I really love about it a Lot of communities are also looking for participants whether they be users or contributors Depending on the kind of community that you're trying to create and Foster you're going to be looking for you know a lot of different kind of people Some communities are already our Sorry, some communities are usually focused on users But sometimes they're looking for contributors. They want to keep that going And keep that contributor base moving and then finally there's also brand awareness and thought leadership if I'm building Open-source projects say around Virtualization, you know, one of the things I might want to do is let everybody know hey We've got a lot of experts about virtualization We'd love to talk to you about it whether you you're working on a similar project or something completely different How can we help each other and that leads to the thought leadership conversation? So these are some of what we believe is happening in these community events So and and how do we measure all that how does that actually work? So in the next section, we're going to be talking about how we kind of meet these challenges And again as I said earlier Kind of from the side so Natalie is going to walk us through some of that Yeah, and like Brian mentioned This is a way to target it We're not going to take on everything head-on These are small actionable ways that we're going to talk about and some examples We're going to give on ways that we're kind of moving the marker closer to having community events be more Measurable having those metrics be easily accessible to us and so The first thing we want to talk about is creating collateral that can be measured and in a place where we're trying to get more metrics Sometimes we need to make something that's going to fit and align with our event strategy That's going to help us move that marker And so the first one that we want to talk about is white papers and to give you an example of What that looks like for us what we've tried We actually implemented a new resource this year as we were looking at our 2023 event strategy we knew we wanted to kind of create something that we could use throughout the entire year At all of our community events to kind of see how people were engaging with us at our booth specifically and so We decided for us this could be something that was really broad It didn't really need to be red hat centric But our emphasis was on the community and so it was how to get involved in an open-source project And so we were able to kind of put that together and it aligned really well with an event that we were getting ready to attend And that was the Southern California Linux Expo more commonly known as scale And we decided that this was the perfect place to kind of showcase this try it out For those of you that are not familiar with scale. It is One of the largest community-run open-source and free software events in North America Can be somewhat similar to a false backstage But here at this event you get a wide range of Open-source experts all the way to a family friendly show that has high school students young families there So you get a wide range of folks that are attending and interacting with you And so this very general how to how to participate how to get involved in an open-source project felt like a great fit also within this show it is In the Expo fall very swag heavy So we get a lot of attendees that are just coming up They want a red hat baseball cap and so to have the opportunity to have a new metric of Someone that wants to dig a little bit deeper into some content into this open-source This was felt like the perfect event and so we rolled out that white paper and We at the show got a lot of interest it felt like anecdotally people downloading it asking about it And then after the show we had a little bit of a hiccup we chose to house this as a github download But then after the show we discovered that we actually could not track our our total download so that Quantitative metric wasn't available for us And so we quickly learned that we were gonna have to swerve a little bit and adjust as the year went on And so then we looked at the possibility of landing pages via QR codes This is something that we had used in other ways We just with scale had chosen to kind of go a different route first But then discovered that it would be best for us to put our resource on a red hat branded page And so with red hat we have the ability we have some tracking tools where if anything lives on a red hat domain We can track usage and so we opted for Upcoming open-source summit Vancouver event that we would move that on to our event landing page Which also had our full portfolio of offerings at the event so our speaking sessions and things like that So at that show we once again rolled out this resource I've got people to engage with us and after that event we learned that we actually needed to make another adjustment We had housed the resource too far down on our landing page And we only got about 10% of our traffic to reach that point And so I'd be fully transparent with these examples. So it kind of illustrates that this is a process It's not just you implement one thing and it's gonna work perfectly for us. It's about moving that marker more and more To have more metrics and meaningful metrics that work well for us And that's one example of using a landing page with a QR code But this is very broad and can work differently for each of your organizations For us it was kind of more of it fall it fell under an audience that was directed towards almost everyone So kind of a what is the project for us? It wasn't a project specifically, but it's how to get involved But with the QR code you and a landing page you could target other audiences You could answer the question of how do I use it and really targeting user races? how is it relevant to them or You can target contributors with that. How do I improve it question of how do they get involved? How do they make an impact in the project? So these are a few ways that we looked at creating collateral that could be measured for us and so beyond collateral so collateral is static and and as Natalie said we can create different kinds of collateral that can target specific audiences and Attract certain, you know needs so if we're looking for contributors We put a lot of how do you contribute content up and talking about our processes and things like that, but beyond Collateral we can have more dynamic solutions. This is actually something we're Looking forward to starting out for the remainder of this year probably into 2024 when we implement it where we're going to have like creating opportunities for active participation and by doing this, I mean that we're going to be able to build some kind of dynamic Channel, whether it be a slack channel or I'm old so a mailing list or I'm really old It could be an IRC channel but something that's dynamic where you can have people come in and Start engaging automatically and That will create some real learning opportunities because when you bring people into Some kind of discussion group and really all kidding aside the channel doesn't matter so much is that you can keep Measuring that channel so we're not advocating discord over slack over You know, whatever new thing comes down the road later We're just saying if you have an active channel of communication You now have opportunities to Engage with that community directly you can bring them straight into your project so in red hats case hypothetically we could say You know from the landing page if you would like to learn more about the fedora project come to their community channel here and We would know that people who are coming into that channel from that particular place on the internet Came from that event and then we would be able to track. Okay. How many people? You know from that event sort of started asking questions about the fedora project and that's just one example of many of course We can have opportunities for onboarding In that if we build some kind of I'll just pick on discord if we build a discord channel and We we have we start fielding a lot of questions about how do we use this project? How do we? Find it. How do we contribute to it? Those are the three basic tenets of onboarding and you can solve that problem by bringing people in and having them kind of Go through a mini orientation session and you just have your existing community members Keep an eye on that channel and be ready to answer questions other opportunities for discussion include Any kind of high-level project discussion like road mapping or you know what the project is? You know going to do next Also best practices some projects have really detailed contribution guidelines and some projects don't And if you don't and you haven't gotten around to writing them yet Channel like this some kind of mini community might be able to give you the opportunity to walk people through and You know creates some FAQs and then boom. There is your Contribution guide as one example you can also do general announcements and events Anything you can think of that you would do in a year as part of your regular community Communications you could do on one of these active landing pages and this is something we're we're going to really start looking at for the future at Red Hat because we feel like this is a great opportunity to start bringing people into our communities from events and now the event becomes more engaged because the conversation that we're having with people will not be Hey, here's a cool baseball cap But it will and here are the cool things we do, but now will also be Here is a way that you can join us And get your feet wet and just take little steps into our community and let's see what happens And that kind of leads us into the next part of our Discussion, which is okay. We've created opportunities for measurement. We've created opportunities for engagement Now, how do we come in and take those opportunities and and build something? That we can kind of feedback into our communities And with that once we're implementing those things we want to make sure that we're measuring all Traffic all the traffic that we can identify So collecting as much data as we can for our audience at those events And so that can look like unique visitors. Obviously. This is the one that's kind of first and foremost We see this at events. You have people come up to the booth. They are a unique visitor You can cross-reference those numbers with the attendance numbers of the events and you get kind of how many folks made it to your booth how How much traffic were you generating in the conference as a whole? That's a great point of metric that you have as well as Engagement at that booth so those people that are coming up to the booth how many of them are taking swag But also how many of them are staying for a little bit longer? Are you engaging in conversation with you're taking notes on? That's a really great point of engagement And then are they also? Taking note of the collateral that you have offered whether it be QR codes whether it be postcards. Are they kind of Interested in the pieces that you are offering them and that engagement is really what's going to drive What your visitor interest is as a metric you'll be able to see kind of what topics are Appealing to your attendees at a certain event or on a broad scale at all of your events What are the topics that seem to be continually coming up? And these aren't just things that are happening in person either your visitor interest can be informed by Some of these active participation groups these slack channels. These are IRC channels I mean you can look at them virtually as well you can see it in your social media presence Are there certain types of posts that are getting more engagement are our people? Gravitating to a certain topic to a certain interest area and these are all forms of ways that we can create metrics To really help us as we're in our event strategy planning sessions And then also to some of the things that we can look at as this traffic is coming in Where are people coming from ideally they should be coming from the local area of your event? So if I did something like this here, I would expect that most of the traffic after the event would be coming in from Spain and nearby EU nations And and so but when you're doing a more you know a Larger event like what we do what we see in North America or or even Asia, you know Where where is that traffic coming from? That's an important and key metric for us because that will help us with localization efforts When we used to do we used to do a lot more events down in Latin America And when we would do that, you know, we would see a huge amount of Interests generated from the audience is there even before we were really measuring it and and so it really helped inform That we had to make a lot more effort in our localization efforts around Spanish and Portuguese language based Offerings Also sentiment and language analysis. This is actually something that my team of data scientists are working on is like Those many communities and active forums whether you're you know something set up just for the event or you're bringing people into the existing community channels for your community or your project rather There are tools out there now that can do general sentiment and language analysis I use them as a community manager sometimes because if I'm looking at mailing lists I may not catch every argument That's happening on that mailing list But if the sentiment analysis shows me that suddenly there's a lot of Not so nice words being used in the mailing list That gives me a signal that maybe there's a heated discussion going on And you can use that in positive ways as well and you can see like Is the messaging that I'm trying to get around my project like I want people to know my project or X Is that messaging being received and our people Talking about it and if they're not maybe they're talking about something else and I need to change my message or something like that And once we've gathered all of these metrics then we can use this data to set goals moving forward And so we can use these to improve our community through refining our events planning I've talked about this time and time again But kind of using that engagement in that visitor interest to help drive where our community event strategy is going And some of that might include building collateral and documentation kind of identifying where our gaps are and then building that to support our event strategy And then also to You know, we're going to take a lot of this data and help we use it to strengthen areas of our community that are weak So we do not have a like this is one of my personal things, but if we don't have a strong on-boarding plan We might find gaps in that plan through this these conversations and the data that's coming in Also, we should be able to if the discussions are leaning more towards the technical side And people want to have features and they want to talk about things to come This will help us start start building technical roadmaps Because basically what you're doing is you're inviting a new group of people into your house And now we're doing it through the events and making that a much more direct stream than the sort of indirect Maybe they'll come by my project later Process that well a lot of us have now and then finally, you know This gets back to kind of the beginning. Yes, community events are not necessarily About generating commercial traffic and activity But for some of us it can't hurt so there might be ways to gently Create more informed processes for like what is you know How do we get people from who have been really using our project for a while to take a commercial offering that we might have? This is deliberately a low priority It's certainly a low priority for the open-source program office at Red Hat But we also can't ignore that it's not necessarily that low of a priority for other organizations so with that we have 12 minutes for questions and also to want to kind of invite you like if you've seen any here anything here that resonates Definitely let us know and also if you thought of something else that'd be like well, I think that's great But what about this idea? We want to hear from you. We'd like to have this be a conversation So if you raise your hand, I'll bring a mic to you. Oh Sharon, you're making me work Sharon makes me work really hard So you're saying that around the community community events, right? So the people that come to the to the boo the stand that you're giving this the swag away with they're all anonymous, right? You're not scanning them like you you know you do at industry events It's not like you're collecting all their data and then sending something out to them. It's all anonymous, right? So we we do scan folks, but we use it primarily just for numbers So we know exactly how many people that we are seeing but it's not used in the same way that Industry events are used to follow up on sales leads Right, and we recognize that too like I will tell you true The like we're just doing headcounts at our booth here with the exception if somebody has a Conversation or they we need to follow up with something like they've asked a question and we can't answer it We will do a note To say they need to talk to so-and-so about such-and-such and then we'll follow up with them later But we've already told them that's what we're doing So I think he had his hand up. I'm just gonna go back So a lot of the metrics you talked about seem really well suited for people who aren't Active participants in your project yet. How do you measure the community building for the people who are already? You know active contributors to your project? Yeah, so one of the things we would do we would start to be more Intentional about like if we host a party for the people that are in our community We would like say how many people showed up and how much money did we spend that's sort of that actually is an easier financial Arranged thing to measure so that would be a way but be more intentional about the community building you have and also How many new people do you attract because that's important too and then how many of those new people that came to your party? Stuck around later and now they're showing up in community meetings two or three months later So being more intentional about those metrics. Yeah, we didn't really address that here, but yeah, that's a great great idea So I got her in the back and then you okay Thank you So I'm wondering I'm I'm running community events and we have We have faced it especially in the last years when we send service after events We don't get many like answers and I wonder if you have tips or advices, you know Or if you run the same issues If it happens with more, you know in more communities So we don't personally follow up with a lot of surveys because it can be hard to get that engagement, but I Know that most companies that I'm familiar with get better Survey Participation when it's on-demand onsite But that after follow-up can definitely be more challenging We've sometimes tied it with an incentive, but we generally we we don't survey after just because it can be challenging to get that Participation, but I'm sure others in here might have had more success with that so one of the main goals that Companies try to pursue when being active in open source is that they say this will attract new talent You know that will apply with the company and and so the sea levels ask me so but is it working? You know you're creating attention. You're giving talks. We're at conferences and so forth. Is it working? Are we getting new recruits and I think yes, so I hear from HR people once in a while Hey, there was an applicant. He said I saw I'm with Mercedes, right? I saw Mercedes as a sponsor. So that's good and that's why I apply and so forth But this just ran him, you know here and there that I hear this from HR departments now a big corporation like Mercedes We have a lot of HR departments and a lot of people and this is not part of the recruiting process So is there do you have any ideas how we could measure that? If it has an impact so yeah, so one thing would be to do What a simple thing would if you like were to take this landing page idea? And just say looking for a job click here and then anything that came Into Mercedes job site you could just say okay Well, who came from that event's website or you build a QR code and bring them in that way a special URL That's coming from that event. So that would be a way to say they walked in the door through here Was that the only reason why they applied for the job? Maybe they've had a friend and they got referred and you know, whatever But that's at least gives you a chance to track them as coming in from your event So yeah, that's kind of what I was gonna say we do tie it to certain event pages sometimes but This might not be the case for Mercedes, but sometimes when we have large-scale openings so multiple openings And we know we're going to an audience at a show that we might want to recruit we bring a recruiter on site We actually do some first-round interviews and such there and then that's an easy way to tie it But that doesn't really help the one-offs and so sometimes kind of having that recruiting place that lives on an event landing page Helps tie it specifically to that event even if it's just the one-off individual that finds their way to Mercedes Okay, so my question it may be a little out of scope for your talk But I'm kind of getting the feeling that your your senior leadership has already bought into community events Like they they see the value of them and you're just trying to do the measure So what if you have an organization that just doesn't get it? How am I gonna measure the impact of going to any event like this or scale or Kcd and so and be able to show them that yes There is a value So yeah, and and one of the things we actually kind of hesitated a little bit about doing this talk We didn't want to give people the impression that red hat wasn't actually Hyper-engaged in community events and that we were starting to second-guess ourselves But like everybody else money is finite so sometimes we have to make choices and And also to we had a bad habit because you might think it's a good thing that we're so open-source Involved and it is but we also kind of breathe this stuff. Do you usually think about how you breathe? No, and we don't either so we don't write this down. I have tons of stories about how we've got caught by that so For you to answer your question Would be I Think start this is kind of why we did this talk is for organizations that are more on the fence or Not even near the fence necessarily and it's like how to You know, how do how do you get those organizations? To recognize value because the ephemeral things like I'm building community I'm I'm building trust. Those are real. Those are valuable things in our world, but not on somebody's Ledger sheet so now we're saying okay if you start treating These people as potential resources and again, I don't recommend you do this like that's your only reason for doing this but that is something that a An HR person an accountant a high-level executive they might be able to recognize that and see the value of growth and collaboration and Then you can show them the intangibles of how communities can band together and pull off things that They might not have expected, but you also have numbers as well so I I I Have a friend who just joined a proprietary software company as their new open-source leader and I kind of Wrote this with that with her in mind A little bit and also with the work that we have to do as well And like how would how can these things apply to? a company like that So and this is why we're saying if you don't have measurable things start building opportunities to measure Any other questions? Okay, I think we're at time So thank you all for coming Natalie and I are actually going to be in the red hat booth right after this So if you have any follow-up questions or comments come by and and talk to us if you have complaints Nice knowing you But thank you so much. We've really enjoyed having this conversation today