 Okay, yeah, we can always switch to that so did anybody come farther than Washington for the conference this year? yesterday we had some people from Spokane and Seattle Any anybody everybody's local past California? where you from? Rancho Cucamong, well, that's a bit of a haul orange Torrance Torrance, okay. Yeah, that's a little bit of a haul too. Oh Thank you for coming Okay What kind of um, are you are you work? I mean tech a tech company in that area? Yeah, I'm always surprised at how much when I drive out that way going either to Joshua tree or whatever I'm always just surprised just how much more there is of development Yeah shopping. Yeah Yep, you have the butterflies right now to everybody noticing the butterflies The painted lady butterflies because of all the rain They've had a real It's a massive amount of butterflies and you'll see them kind of fluttering through like a little train Okay, thank you for joining the mentoring track. We're gonna go ahead and get started with our speaker Hello, I Understand now why they were saying that this was interesting Okay Folks can hear me right So welcome folks, thanks for coming out to the mentoring track, sorry couple minutes late I Believe I'm a technology guy, but I can never get the hardware stuff to work. So I'm really thankful for The folks here at scale. My name is Karthik. I'm from Austin and today I'm gonna talk about some of the stuff that we do in Austin with respect to the tech community over there Draw some parallels and see we know what are some of the best practices that we've learned over a length of time So I actually work at Oracle when I say that I'm actually not a database person Sorry, I do know how to like do one normal form and two normal form But I find that the questions I get asked are very complicated. So if you do have DB questions I could find somebody inside Oracle to help you answer that if that's relevant I do more cognitive stuff. We actually my team got acquired into Oracle. So I know a lot of stuff about containers Containers and all of that the other stuff and I used to be a developer on the managed Kubernetes team Before Oracle, I kind of like did a whole bunch of stuff Most of these companies are based in Austin or you know, they were like remote kind of gigs But in general like I like building You know stuff with friends of mine the Austin community is pretty big So we can make connections pretty easily I'm a maintainer on Gauntlet, which is like an open-source security scanner And I also help run DevOps days often Container days and Cloud Austin, which is a little user group that we have over there Do a bunch of other stuff, but you know kind of all the same Quick shout-out to my corporate overlords. I like to say So Oracle is doing a big thing with like cloud and whatnot and there's a lot of confusion in the enterprise space about cloud native and what that is So we have like a little site cloud native to oracle.com has the solutions blog posts and whatnot Feel free to check it out. I made a quick link bit.ly.com. I heard cloud I heart cloud So check that out if you want to So we're here to talk about community or I hope it's not just me talking about community I actually have I tend to like do a lot of slides, but only have like 40 or something slides I'm gonna try and finish early and so community in my opinion is like a very collaborative thing and so it's not like One person doesn't make a community. It's kind of like, you know a group effort and People who are like like-minded kind of come together and like talk about stuff And that's how you built, you know, that's how you kind of like build it build it then kind of grow it So from like a high level The Austin Tech community today. It's like very very vibrant we have like You know a bunch of conferences that come that are located in Austin, but also come to Austin serverless days Longhorn PHP conference That happens in May DevOps days, which also happens in May Day-to-day that happens like early in January last con which is security conference happens in October B-side just happens in a couple weeks Keep Austin Agile which also happens somehow like May is a really popular month for Austin conferences Because I don't think we believe in the rest of the year, but for whatever reason will I do everything in May? So yeah, keep Austin Agile happens in May as well. South by is like our big You know big conference where like, you know, Michelle Obama and stuff come to it to like speak At that and then finally in a tech Austin, which happens in the fall. There's also a bunch of meet-up groups I was trying to get like a number of meet-up groups But like it's almost impossible because there's like six or seven meet-ups that happen every day From Monday through Friday, and so literally any day of the week you'll have like some kind of meet-up going on Austin Tech events is like a pretty cool thing that it's one of the like side animations that I like to talk about I didn't do it. It's a bunch of folks that are in the community that were like Hey, you know, there's a lot of meet-up groups out there and there's no like good information sharing between all of these groups so maybe we should start like a you know weekly newsletter and You know like a Twitter page and it has like, you know 11,000 followers and they do this not just for Austin So they do this for like a bunch of different cities that cities as well. I'm not sure If LA is a part of those but you can ask them if you'll want to do one for LA too It's just like a bunch of four or five people So I actually run cloud Austin, which is a meet-up group It's kind of an older meet-up group now This looks really cool. In fact, our logo is brand new like this is something that I made up Think like last month or a month before It's just using Stock photos shouldn't touch this because it's volatile stock photos and some fancy paint work But the meet-up group itself It's been rebranded once before it used to be Austin cloudy as a group. We call it cloud Austin today What was it? It's like the Facebook movie, right? It's called the Facebook before and they're like, no Just called it Facebook. So we're like made a four letter thing into just a simple two letter thing It's over 10 years old at this point in time But and it has like 2,600 people that not everybody comes to it every month But we have like a steady stream of like at least like a hundred people that come out to each each user group We also kind of do like two talks in a specific meet-up So like if the first talk is kind of boring you can be like well the second talk was interesting so like you know it's We have like multiple things that kind of go on and we kind of mess around with the format talk about that a little bit later on but Today, you know often Has kind of grown a lot over 10 years Personally, I've stayed in Austin. I think it's like 13 years right now We call it like Silicon Hills Which is like cool for us and then I was reading this article the other day And so there's like a hundred and thirty eight thousand tech jobs in town And I think Austin only has like eight or nine hundred thousand people. So That's like, you know one ninth for actually more than that With number of jobs for for technology folks 31 billion dollars go into Austin's or the area is over economy overall economy and 35% of that kind of comes from technology gigs, which is really cool Apple is also we lost the fight to have Amazon HQ to in Austin kind of like one of the towns that they were looking at But Apple decided to expand in there and also the Army Futures Command is built HQ in Austin, which is all kind of cool But like things were when I started my career and often was like very very different 15 years ago You know these were like primary employers in Austin so like with people that kind of Lived in Austin and like, you know, you'd meet like friends like friends of them to whatever you'd be like Hey, what do you work and be like it was either a Delaware IBM and if it wasn't Delaware IBM It would be you know, Nationalism in Samsung EMD There were some folks that work, you know for So the local government kind of runs out of Austin. So there's a bunch of like tech stuff that go on there So some folks used to work for or still work for like the local government UT also has like a lot of like tech stuff going on. So some folks work there and Bleeding edge folks back in the day used to work either at bizarre voice like Rackspace or indeed And there are a couple of other companies But like a lot of times you'd be like, you know, when somebody used to leave an eye We used to joke about it where we say hey your transition path is like you start as a programmer analyst You go one two three and then like you end up at bizarre voice or indeed or something like that So it was kind of like a little joke that we had in the community and From like a user group setting like There were a few there were few and far between in terms of user groups There were like some stuff for the language developers So if you were a Java developer, there was a Java user group and I tried to trace this back I did some like history stuff The other day so I tried to trace it back and it actually started in like 2001. I see I saw references to that dot-man and php also kind of been around for a long time in terms of use groups in Austin and then oh, which is the I forget what the oh was Full form is but it's a security group That started in 2000. Well, it's been around but the Austin chapter started in 2006 and you know the ecosystem in Austin is really interesting because like the different groups so like developers kind of talk to Operations that talk to security. So it's kind of like not three different siloed kind of groups They're all kind of like blend together in some ways, which is really nice And you know we we can kind of like work together in a lot of ways to solve problems Also user groups used to be a thing inside of companies So this is something that you might see in large companies where it's like hey, you know every every month You might have like a speaker from the company kind of present, you know technical things, you know inside of the company So like we had Java technical exchanges. We like to call it used to happen every month, but It would be you know for You know engineers or program analysts inside the National Instruments And we talk about like different like Java things that we kind of playing with or whatever Much like you we do today at a meet-up by just like an internal kind of setting so oh Yeah, and apologies for all of the so all of my slides have cats on them and I did that because I'm like an enterprise guy now So like I can't have like cool slides So I was like, you know what I'm gonna put cats on every slide, but they're mostly just like background kind of things But you know every That I think that thing that really changed was about ten years ago Everyone started doing a whole bunch of stuff with the with AWS and they were trying to figure out well actually not ten years Yeah, like a little over ten years ago They're trying to figure out what the heck this was Whether this is actually gonna be a thing or whether you know it was gonna go anywhere or whatever and so like we found ourselves in a same kind of team where it's like Our task was to build products that we could host on Amazon both services So we're like, okay. Well, you know, let's go and figure out what You know EC2 is let's go figure out what S3 is blah blah blah working with those APIs, etc and Back in the day like we met some folks at a different company called pervasive so We had a team of national instruments And we met these other guys at a conference from pervasive which is also like another company in town and we're like Oh, you're doing AWS stuff. Hey, we're doing AWS stuff, too And then back in the day it was like AWS is really simple because it was like S3 You know EC2 and a couple of other things like before RDS before all like all of that stuff So it was really simple and so we had like questions on S3 API So we're like chatting with them and whatnot and so or in training emails So we were like, okay. Well, you know, we were like kind of helping each other out. So we like formalized it and instead of like doing stuff over email, we were like, well, let's meet for lunch and So this is actually like a picture of the cafe that we kind of met at And you know, we were like, let's go to lunch and it was still kind of awkward because like not everybody on Their side and you everybody on our side and we're all we're all engineers, right? so it's like Sometimes it's kind of like weird to meet new people and like talk to them and you don't you don't want to feel judged Like you don't want to feel like hey, I don't have any questions or I don't have any answers I just have questions and sometimes it's like a little bit awkward But so it was like it is kind of awkward because we were sitting there around round table I think there's a couple more folks that I don't like that were at that first like at first lunch I just don't remember who they were so I like I didn't add pictures But there are two more people But we were like talking about stuff and like all of us just had a bunch of questions We didn't actually have answers and we're like, well, you know, they do you know how this API works blah blah blah And we're like, I don't know and so we're like, you know talking about stuff a little bit more And then we're like, well, I wonder if there's actually somebody with answers To a bunch of these things. So we're like, maybe we should like start like a meet-up group And you know, we because like you're from a different company. We're from a different company Maybe there's other people kind of doing the same thing. So we're like, okay Well, let's start like a meet-up group. And so back in 2010 like way way long ago We started this group and like I did this search back and I found like a bunch of tweets from way back then We called it a few G back back in the day. So like this guy Bill He was I think the director of marketing at pervasive and pervasive actually had pretty big Pretty big space that we could use so they were like, hey, you know, we can just do like a meet-up here It's okay if it's you know, it's kind of like the room like this But they were like, yeah, it's okay if there's like 10 people in there. We're just talking, you know technology So it doesn't matter So bill is actually instrumental in like Getting us through the workflow of like how to get room and whatnot Tristan He still lives in Austin Works, I don't know where he works anymore, but he used to work at buzzer voice And he had a lot of like technical knowledge. So back in the day people would like asked Tristan a bunch of questions A lot of folks already know Kote and Michael Jordan Kote works at pervasive art. I'm sorry. He works at Pivotal right now But back in the day, he used to work at Dell and then James Wicked He does a bunch of security stuff in the day But he was kind of like a big champion of on the security side in Austin He actually helped start the last con conference and I was also like did a whole bunch of stuff for OAS Boston So it's kind of interesting to have, you know, his perspective to help build a meet-up group And so we're like sitting in this in this room kind of talking about, you know technology And we're like, hey, this the first meet-up actually went really well. I Think Michael Jordan says Yeah, he says there were like 60 people that showed up which is kind of big because we were expecting maybe 15 And so when 60 folks showed up we're like, whoa, so a lot of people are doing cloud stuff I guess maybe this is gonna take off So it's kind of like the consensus back in the day And so our journey kind of started there and then you know from a tech standpoint We're using like Google groups because Google groups is like the coolest thing back in the day We use that we used Eventbrite to kind of like help people sign up and find out more info Nobody use meet-up Because meet-up was more for like use meet-up to find like tennis buddies or use meet-up to do Sports kind of stuff like you never used it for like technology and it's like completely different today But we have this like one transition from like, you know, this technology stuff Where we did Google groups Eventbrite and then we're like, oh, we should do meet-up as well And then we're like, oh, maybe we don't need Eventbrite. We'll just use meet-up And then finally we're just like everything goes in meet-up So initially it used to be like random whenever we had topics and then we kind of formalized it into monthly meet-ups And we also one of the cool things we do is like a festive thing at the end of the year where we do like a 12 Clouds of Christmas so they're like five minute lightning talks that people come and present And every year it's kind of grown. So at the at the start I think that 12 talks and now we do like 20 talks or something But it's really cool because like the whole room is packed and people just kind of show demos that a five minutes long So like the barrier to entry is not really that large But you know, like I'm talking about all the positive stuff, right? And you know, there are things were like not always like super happy and You know, things kind of fell apart every now and then one thing that happened a lot is like people will be like I will sponsor food for your meet-up and food sponsors are Super important because when you're trying to get something off of off of the ground Sometimes people don't care about what you're talking about. They're like more in it for the food or more in it for the swag So when like a food sponsor like yes, I will help you bring pizza And then they're like, ah, you know, yeah, we can't do it And they tell you like a day before or like the day off it pisses off people quite a lot They're like, man, I came here you promised me pizza and I'm here and like I don't see anything And now I'm like now I'm cranking and I'm hungry and I don't care about what we're gonna do. So I'm gonna leave So that that happened quite a bit early on But you know, it's gotten better over time. But you know, that was kind of one of the issues And one thing one incident that we learned a lot from was this Jeff bar road trip So everyone knows most folks know who Jeff bar is he works at AWS You know, he kind of like does all the product announcements and stuff like that Back when it was just, you know, it was just kind of him doing the product announcements In 2013, he was like, he was like, I'm gonna do this crazy trip where I'm gonna start like way in the east and go all the way West all the way west to Seattle I'm gonna do them in a car and stop on like different cities along the way So he was like, I'm gonna drive through and so he was like, let me know if you want you want me to come by So like I sent him an email and I was like, hey Jeff, you want to come speak at this user group in Austin? And we'd love to have you so he was like, sure. I'll stop by because after drive through Houston anyways and an Interesting thing happened because Jeff at the time was like the biggest speaker we ever ever had So like really big name Really influential person in industry like comes to our little meetup group which on average, you know we'd have like 30 40 people come every month and so And we were doing our stuff and like event bright and meet up, right? And so we're like, okay, we'll do this thing. It'll be awesome. It'll be really cool content great And Jeff is a really cool guy but the thing we we realized was you know There was a lot of demand from people wanting to come to it so we're like, okay well our we can only accommodate like a hundred people in the room and You know, we'll set the RCP still hundred so like pretty early on we were like, man, we're gonna blow this number And so we talked to the facilities that pervasive and we're like, okay Well, like what can we do and they're like well the fire marshal code is 120 so we can add 20 more seats So we'll add 20 more, you know seats to the room, but it's gonna be like really packed And it's gonna be, you know pretty pretty tough to have like everybody in we're like well Still he's like a really big names to be there. So let's increase that number. And so we did that And sure enough like within the day of those like 20 seats are gone Because 20 more people are repeat and we're like, okay Well, let's start a wait list and we had a wait list of like 40 40 something people and Then we're like well our hands are tied from an organizer perspective because like we can't break fire marshal rules But at the same time we're like well, we don't want this is like one of the big things that's happening in community So we want people to be able to attend But we were like well, you know, we tried our best So we were like, okay, you're not gonna come, you know on RCP so somebody can take your spot blah blah blah But in the end what happened is like a lot of people were really upset They were like, well really want to go to this thing We have something cool going on in town and you're saying you can't we can't come to it And what happened what was even worse was That day or that evening we had like a weird weather thing. So like it started to rain I don't know what it's like in LA, but in Austin if people if it rains like nobody nobody goes anywhere So it rained and sure enough like I think totally we had like 75 people show up So we have like, you know, it's like I'm it's still pretty packed But there's like open seats everywhere So then people got even more pissed off because they were like I RSVP to this I would have come to this thing, but you said I couldn't come and you know, it's not not even a full room so it's kind of like some of the flat that we got but In we kind of like as an or group we kind of talked about it for a while We're like well, you know one one thing maybe we could have done better is made it more like Made the process of like the stuff that we were doing behind the scenes a little bit more transparent So we ended up sending like a letter to or not a letter but like in our meet-up thing We sent up sent out an announcement to everyone kind of like talking through apologizing for You know what happened, but at the same time kind of talking through some of the stuff that we were trying to do from a silly facility standpoint and also like how to kind of smooth stuff over a little bit more But it was kind of a train wreck, but that was like one of the things of all the things that's happened in like the 10 years This actually taught me a lot from a visibility standpoint There's like people are more forgiving if you kind of like talk them through the problems that you are running through versus just kind of keeping Things very, you know saying saying no, but not really giving reasoning behind Why you're doing what you're doing so I was kind of eye-opening and that's something that I kind of take today Even with like Delos is Austin and like other conferences that I help organize So today, you know, like we've grown quite a bit. We have like 2600 folks where we always kind of do With third Tuesday of the month we always do our stuff at Rackspace and You know, we'll talk about some of the reasons about why we do what we do and why it's like Oyster Tuesday always that assistive venue and we we try and get food and drink sponsors whenever we can From an org perspective we've kind of grown so it used to be I think like 304 folks, you know, a couple folks in pervasive a couple folks that help kind of like, you know, get speakers and Do like all the meetup management and whatnot, but we've grown to an org of Like eight or nine people where we have some people doing venue stuff some people doing You know trying to get speakers some people doing meetup management some people talking to You know a lot of sponsors end up sending us an email via meetup or whatever So, you know managing the communication from that front so it really helps when you have, you know A lot of folks kind of involved But one of the issues when you have a lot of people is you also end up You know trying to do the same thing over two people are trying to do the same thing Which is normal like large teams, you know, there's replication or duplication of effort ends up being a little bit confusing so We we used to do everything via email, but now we've kind of transitioned to do a lot more over Slack And it's a lot more efficient because we can say hey we you know Here's what we're doing like here's our ideas and everyone can kind of chime in From an organized organizer perspective well to do a little bit of like the Zapier integration with Slack And we have Instead of you know people emailing us because we're this is kind of like not our we're not paid to be organizers It's just kind of like a side thing that we do for the community So we're like well, you know It takes a while to like respond respond to emails and whatnot So we have like a Google form for if you want to speak at cloud Austin There's a form that you can go for a lot and you know, let us know Sponsoring is kind of the same way now and we do this integration with Zapier into Slack because we're all about Slack So this is kind of like an example of that words like the bot tells us Hey, somebody wants to do and talk on whatever and they want to do it for this month or somebody wants to do a sponsor for this month, etc. I Like high or I got rid of some of the sensitive information there And yeah, so we kind of do that So wanted to talk about like some of the things that I've learned over time because I think that's the most important part From a user perspective right there a lot of times the reason people come To a talk or come to a user group is because they are really interested in what somebody has to say from You know presentation standpoint demo standpoint, whatever so like if you do a presentation on Kubernetes today There's like a lot of people that were like hey, I really want to understand like the nuts and bolts to benedicts. I'll go to this thing so If you're an organizer, you know, you have to realize that content is like the number one most important thing for why people want to come to a specific event and This is like it's true not just for tech, but you know really anything out there If you have like a You know really good speaker really good really good event people will be like I really want to go to that thing because I'm interested in it On the flip side You know we in the dev specifically like devop security space There's a lot of vendors out there and vendors really want to talk about the product And I think that's a good thing But it shouldn't be like the meat of your the talk because like People don't want to sit through an hour presentation of somebody giving like a pitch deck or like here's how my product works or whatever They want to hear more like generic information. So one thing we tell One thing we kind of help our speakers with is like hey We don't want a sponsor pitch or like we don't want a vendor pitch So we don't want to see like how your product works But instead talk about you know some of the technology in your product instead Talk about engineering problems that you have talk about, you know, how you might have issues like You might have had issues like taking your product from like one cloud to another or whatever So like always kind of send out engineers to speak about What you're talking about because our audience is specifically more like engineering folks So they're like developers devop security, etc so we always say no to that but you know from One thing that sponsors kind of like is we help them we let them sponsor stuff. So audience attendees love free food So like it's you know, especially when you're kind of starting off And let's say you're building like a meet-up group today It's great to have you know pizza or something like that because people people will come for the food Especially in communities that you know what that don't have a lot going on and there's not many choices I was like well, I'll like go for the food and might listen to something interesting to and then later on there Be like I will come for the content. I don't really care about the food that thing switches But one of the things that from you know transition from the last slide is You can kind of say hey, you can do a vendor pitch But if you sponsor you can do like a five-minute demo or something like that And so that kind of helps with you know a getting getting sponsors in Satisfying their need to like demo something but at the same time also, you know providing food for the audience, which is great The other thing I've learned is like just transparency so just be transparent of your process of like the stuff that you're doing behind the scenes because You know if you're not then people kind of like question what what you're trying to do or things get a little bit confusing and then you have to You know kind of step back and do things differently, etc But it's always more convenient if you're just like we're doing this thing And meet-up is really good about you know having the comment section So one thing we have a lot of issues with recently is and it's kind of the same issue that I talked about in the Jeff bar thing But you know we always end up hitting our attendee cap So we're always like hey if you're not gonna come to the meet-up You know on RSVP, so I end up like actually sending that message Almost every meet-up it feels like now because we always end up hitting that cap But on the flip side of that is a lot of people we also have like a meet-up If a hundred people say that they're gonna RSVP to come You know we'll only have like 75 75 of those we have like a 25% drop-off There are other meet-ups where you know you have like 50% drop-off or whatever It just depends on on the group and how how core you're like How core your members are basically? This is really important it's something that we sort of forget from time to time but have a coat of conduct and I say this because You know a it helps From somebody new wanting to come in wanting to be a part of your community They kind of look for that because they want to know if like that's the space that you are providing as an organizer is a safe space You know whether they can come and like You know actually be able to Participate and not feel threatened on the flip side of this is you know, I've seen like weird things where You know you might have like happy hour or something like that and somebody like drinks way too much and get Gets really belligerent and then you're like well if at that point in time you're thinking what should we do? And what's our code of conduct? It's already kind of late. So you you rather You know things through some of those things up front have a code of conduct for or like a playbook of things that you might want to Be doing, you know in those times of emergency Versus, you know thinking about it on the spot. So Something to just kind of think through also How many folks are like introverts in the audience? So a lot right and and I am too like even though I do a lot of public speaking Actually kind of like after a while and like man I I would like to just like sit in my house and Talk to my wife has three cats. We have two dogs. I used to be a dog person, but now I'm a cat person I'm like, I just like hang out with the cat or hang out with my kid. I'm like, I don't want to talk to people but one of the hardest things that for me As an individual like going to meet up there's like I hate going alone And I was trying to think through some of the stuff like so, you know, it's not just like tech stuff Right, so if you go like let's say like you two is coming to town, right? They're like, man, I really want to go to this concert, but I don't want to go alone I'm like I would rather go with like a friend of mine like family member or something like that So most of the time you're like, you know, you're trying to go with somebody instead of like going alone so we have this thing like we we call it the buddy system because Tech, you know naming is hard But you know, I'm always like, hey, I found this really interesting thing that I want to go to And I have like a close circle of friends. So I'm like anyone anyone else going to this or anyone want to go to this And so the likelihood they've done a bunch of studies and the likelihood of like two people saying, okay We'll go to it together is like much much higher than a single person saying yes and then dropping off later. So If you're like trying to you know, get more people to meet up or something like that be like, hey You know bring like come out to come out to the meetup and bring a friend because you know You it's like if you're kind of sitting through something and you don't like it At least like you have, you know, somebody that you can kind of commiserate with in some ways Or you can be like, yeah, you remember the thing we went to you man. That was horrible But I'm glad I was there to like share it with you The other thing I really like is just gratefulness for organizers also volunteers Because a lot of things that we do in Austin specifically, they're all like community-based. So it's not like we Nobody gets paid to do all the stuff or just like we're doing it because we believe in like the general community in town So just like gratefulness for like for folks organizing stuff and whatnot Because organizing is painful because you have to get a bunch of stuff in place. What not make sure logistics are taken care of and You know, a lot of times people You know, I take time out of work to do it's like some people have families But they're like well, I'll go to this meetup thing because I organize it and you know Takes time away from whatever else you might be doing at that time So if you're like I practice this a lot But if you're an attendee and you go to a conference and there's a lot of volunteers or whatever, you know always kind of thank them for The effort because they could be they could be somewhere else doing whatever else but they're there to like, you know Help you out in some ways. So You know, that's just that's just the thing that I do a lot and then For any of the folks here, like are you all organized this for meetups already? Or like quite a few, okay? So some tips that you know, I've kind of learned over time is Fix venue fix date and time because what happens what happens a lot is If you have you're trying to build a quorum of people Want to come out for something on a regular basis if you're trying to build, you know a set of like a core team basically It's easier for somebody to be like, oh, yeah, you know, this meetup always happens at this location And on on this on like this specific day, but so they're not like Where do I go? Like is it in the morning or is it or is it in the afternoon or is it in the evening? And then which location is it at so like there's no like Flow chart that their mind has to go through to figure out where it is It's just always the same thing. So over time, that's really useful because they're like, oh, yeah You know, it's like third Tuesday. Oh, yeah, you know six o'clock It's that that's that rack space has a thing for cloud Austin. So it really helps with you know getting a gathering momentum from from a tindi perspective and then your attendees actually you really help out too because they're like Oh, yeah, you should go to this cloud Austin thing and they remember that it's like third Tuesday Third Tuesday at rack space or you know, wherever whatever that venue is Also understanding what folks are interested in it's super important Because your attendees is your audience, right? So you kind of need to know what they're what they're wanting to learn Or what they're trying to get out of a meetup. So one question that we always ask is We also ask it in our in our form when new people sign up for the meetup Like what what topics you want to learn but also at the start of the meetup We always kind of do a show of hands of like how many people are new And you know kind of say hey, well what what interesting do you to come out to the meetup so we can kind of age generate conversation because a lot of me that meetups are about conversation rather than know somebody presenting whatever and I Lost my train of thought Okay, well Yeah, and then experimenting with formats really good too At the start we always used to do like one talk an hour long and you know like We have like food upfront so like people kind of eat for a little bit And socialize then we have the one talk and then we end And then we were like well, you know, sometimes talks were an hour long Sometimes talks were an hour and a half long So we're like the we started to notice that the longer the talks the quicker that people kind of zone out And end up leaving beforehand. So it's easier to We did an experiment where we're like, okay What if we did to like to 45 minute talks which are like 35 minutes ten minutes for questions And then transition to another talk. So we sometimes do that. We sometimes do Like lightning talks, so I talked about this file clouds thing that we do every year at the end where we just do, you know Two hours of lightning talks like couple of times a year where they're like five minute talks It's kind of like whatever topic you want to talk about And so the format's a little bit different from that perspective other times We'll just do open spaces where it's like people will come just to kind of talk about a specific technology things that they want to talk about and whoever is in the room kind of like will you know You know join the specific group to you know Speak about whatever that they want to that they want to kind of participate in so it's kind of an interesting format DevOps days conferences kind of You know the group of people that you have is the group of people that you have So that's kind of like an experimentation thing that you do over time But don't experiment too much because then they'll be like yeah every month It's a different thing so you know be consistent for like a few months and then change it around for a month And then go back to a plant that kind of works just to kind of keep it fresh in some ways be inclusive because You know we're all we all come from like different backgrounds and we are all kind of interested in different things so It's important not to like shoot down people's ideas Or you know things that they might be saying so inclusivity is really important and at the same time, you know have a team because You know organizing especially if you're doing it from a volunteer volunteer Perspective is it takes a lot of time and one thing that if you're an organizer like the I've seen a lot where People will do like a meet-up group and it'll be really good for like, you know six months Then the person running the group would be like man. This thing really blows. I'm gonna stop doing it So like groups kind of come and go And it's really hard to you know kind of keep momentum As you go along and you just honestly get tired like you know your things in your life might change Or you might just be like, you know, I'm over Over whatever You might get jaded etc. So you know you kind of want to watch for Burnout I Notice this like for myself like a couple years ago where I was just like I stopped kind of caring about like tech stuff for a little While and I was like I don't want to organize this called awesome thing or I'll organize it But I don't want to go to it I'd rather just stay home and then I was like what's actually going on with me So I started, you know like talking to other folks about like burnout and You know, you want to If you if you catch it early on you can make you can make changes quicker versus, you know Realizing you're like totally burned out and you're just like man, I don't I don't want to do any of this stuff one of the tricks behind it is to get a team so you can kind of You know do it together so it's not just you so it's not just like a Road that you're walking walking alone on it's you walking With other people as well. So it's a lot more fun easier to like distribute the effort so you can kind of do things you're really interested in like the The thing I showed with Zapier early on the integration. I really like automation kind of stuff So I was like, oh, yeah, you know What if I did like an automation thing and put this thing into slack and built the bot or whatever I was like able to kind of do that and like it was really useful for the group for the organized group But at the same time, I think like a few years ago I would never done that because I'd be like like oh my god We need a speaker and that's the finding speaker is like the most important thing so you can kind of you know Do things you really enjoy And it's also important an organizer. You find yourself in a position of power And so, you know, it's important to be able to give people opportunities So whether it's like speaking opportunities or kind of like You you end up like Learning a lot from you know sponsors you will end up learning a lot from peers Speakers attendees, etc. So you can kind of make connections a lot And that's something really powerful because you know, sometimes somebody might approach you like hey, you know, I just got laid off Do you know anyone who's hiring and chances are you might have met like the sponsors a month before and they're actually like hiring You should be like, you know, you should talk to this this other person or whatever seeking help People help make people bridges basically um And yeah, so like, you know over time like Austin has grown like super friendly a lot of folks just kind of Enjoy kind of talking to each other And so it's not it's not it's it's a lot more Inclusive and a lot more open Then how things Used to be when I first moved to Austin And then you were kind of like nervous talking to somebody else because you didn't want to be judged But the sense of like judgment who has really kind of gone away It's still there, but like it's definitely diminished quite a bit over time I said tech folks kind of know each other ish because I think at the end of the day we're still introverts. So sometimes it's hard to like Go and like meet someone brand new But you know in in general, uh, if you know, if I have questions on like Jenkins or whatever, I'd be like, you know I know eric. I know eric long he could probably answer this question for me Or if like if I have questions on kubernetes or whatever like I'd be like, you know, I think paul zarkowski might know this or In fact, uh, the at the ibm booth today, uh, jj is working there and jj is from Austin So like before this I was like, hey man, so I went and I talked to him for a while Because we all kind of come from the the same place and also You know make it like a culture where it's okay for people to ask questions And because we I think this gets worse as we get older in our careers where we're like It's bad to ask like simple basic one-on-one kind of questions because that's how we're kind of molded And you you you don't want to feel like you're asking like really dumb questions But it's okay. Like, you know tech stuff changes like so uh, so quickly, you know when Nobody knew about cloud like 15 years ago Nobody knew about containers like 10 years ago like nobody knew about kubernetes five years ago So like tech evolves and changes over time. So, you know, it's okay to ask like Really simple one-on-one questions and you know not feel not feel judged for it. So as an organizer, it's like You should kind of breed That kind of culture in your groups So with that, um, I don't have anything more I just wanted to say it's like communities that really open things you as an I as an organizer or you as an organizer you help like foster that community But in the end it's like kind of hard to control how those things kind of grow Sometimes they're really fast. Sometimes they're a little bit slower But in the end it's like, you know, those are the people that you end up driving with so That's what I have. Uh, folks have any questions or anything? Um Happy to take them Thanks What's up? Thank you. So We host a meetup monthly and It's pretty awesome. And I think it's a great thing to keep going The one thing we always have a really hard time with is sourcing speakers. Obviously you mentioned content is Critical and I totally agree. You get it. We've had some really great speakers who've had a much better draw than others which We want to keep that going. How do you suggest? Continuing to find great speakers in your area and you know for the qualification I'm in the bay area. So of course we've got like tons of awesome talent around us for so Do you have any tips on how to do that better? Um, actually like what what was your meetup that you organized just to shout it out? Sure So we host at twitter on the third wednesday of the month. It's the san francisco reliability engineering meetup We're on meetup. Uh, you can also Come find me. I'm my twitter handle is mr. Mocha spell out mr I'll be talking in the next session next door too if you want more detail Nice, and uh, yeah or come find me. I'll give you my business card. We'll follow each other or whatever Won't be creepy at all. I promise So, uh, but yeah, if you're ever up in the bay area, like I said, we have a great meetup and we're always looking for speakers So we had a great little community coming together so far We're running it. I want to save for like three or four years now We just kind of take a short hiatus every holiday season around december because nobody's around that month But the rest of the time we have a pretty good following usually So come join up Basically like what what I'd recommend is and one one thing that we do is we If there's so I go to a lot of conferences And so I'll be like if I go to a conference and if I find out that the speaker's mosque or whatever I'd be like, hey, uh, you should come present this the same content that cloud user group or at cloud often So that's that's a strategy. We kind of use Sometimes we'll hear somebody doing really interesting work Or, you know, we have an org of eight eight people. So it's kind of like federated a little bit more So somebody would be like, you know, like have you seen what Tristan's kind of working on these days? It's like totally blows my mind. So we should get him like out to like talk about that stuff You know Okay Right. Yeah, and and that's the other thing like it's it's if you kind of Spread it out. So it's funny. Um Ernest and I so Ernest is one of the other organizers for cloud Austin Like he kind of is like the main guide that runs everything But he and I used to work on the same team at national instruments like 10 years ago But like now we're kind of like doing our things and you know, I'm more interested in containers. He's more interested in security Um, so sometimes, uh, you know, you'll meet people that come to the meetup And they're always there like regardless of what the content is is they're like your core followers And sometimes it's good to be like, hey, you know, you want to be an organizer with us Like, you know, come join come join the team kind of thing Um, so that's kind of how we ended up growing out because there there would be a lot of folks that would have really good ideas Or tell us like I went to, you know Jenkins rolled Last month and somebody spoke about this thing and I think they're local You know, let's let's have them out if you can and so Sometimes we'll just like follow them on twitter and be like, hey, would you like to come present this thing? You know at the local meetup or you know, I'll send them an email or something like that And as an org perspective, you're just kind of like, you know, trying to make the connection and have folks come out Um, so that's that's an idea the one. Um, I don't know how it works in the Bay Area But you mentioned how a meetups kind of take hiatus In the wintertime actually Austin is the same way the 12 clouds thing that we do It's not just our meetup. We have like a whole bunch of meetup groups Kind of join us for that one. So it's like six groups or something like that that come and like present on different topics so like The postcards meetup, for example, we'll we'll collab with us and we'll have like postcards content. We'll have You know cloud content, uh, google gcp kind of content or whatever So it's like a collection of stuff because it's all in the end like people just want to hear cool things So they're like, okay. Well, we'll like talk about that stuff So long wind and answer, uh, and then if I think of ideas, I'll I'll can treat it at you as well What are the questions? All right Five minutes. Okay. How do you deal with nervousness going to new places? Is that one reason why you should always hold it at the same place as well? Uh, nervousness you mean from a speaker perspective not just from a speaker perspective from even an attendee perspective Um from an attendee perspective. So one thing we do Which I think is like a good practice is um, since I go to all the meetups You know, I know who's kind of been there before who's new And so for the new new folks, I'll be like, hey, you know, I kind of like go a chat of a conversation like what's brought brought What brought you here? You know here's kind of format or meetup, etc. And it kind of you know helps with New folks coming in like, oh, yeah, you know, this guy is an organizer But he actually came and like talked to me before At the start of the meetups. So like, you know helps with uh with that From a speaker perspective, you know, like meetups We actually Are like as an organizers We're very inclusive So like if somebody is like struggling to like talk about or whatever we like encourage them because I mean it's just a meetup It's not a it's not like a big conference or something like that where you have like, you know, hundreds of people You know, it's pretty intimate. So we're like, uh, you know, it's it's fine if you screw up or whatever So it's not the people for me. It's the new places that really get me nervous I don't know if that's that common. It seems not to be quite rare. Well, did you have something to add? Well, let's get you a mic real quick. So it's because we're recording all this stuff So I have the same problem. Also, I hate going to new places and I'm really bad at following my dps And uh, so I'm always getting lost. So when I do my meetups, I go ahead and I post Kind of like gps Images like I'll look it up on google maps and show you like, hey, this is actually what the building looks like From the outside and I'll go through and I do give people like detailed information Because the gps will tell you like how to get there But sometimes there's one weird way to get into the building that will stress people out or something like that So I try to be as detailed as possible When giving that information because I know that's something that prevents me from going to new places Yeah, that's oh, yeah, that's a good point and parking because parking sucks always yeah We do that a lot for dev ops days as as the conference and we've been doing this meetup at the same place for like Six seven years now But for a conference stuff we definitely do that where you have a lot of signage and yeah I used to be that guy who'd like drive the day before Or like a couple of days before make sure that you know Like you park in us in in the spots parking thing was open kind of thing Make sure we have sign it so because It used to be kind of a walk So it's like make sure we have signed just specific spots so people don't get confused And you're not like man. I don't know what the hell I am. It's like it's really it can be a really nerve-wracking From a user perspective. So um Those are I think all of those are really really good tips Um anything else did you have a question to you? Yeah So I I know you said that you have the event at the same location every single time. Um, do you Do you ever run into our Or look into doing events at different spaces to get more people come to your meet-up because in la like it's very large So I have to when I'm running my event Try to plan different venues so that we can reach all of our members Um, so I just wanted to get your viewpoint on that. Um, we we sometimes have done that where we'll be like, okay, uh, you know Those events a while ago and this was like a few years ago where We had like a doing cock croft come to speak in austin and so we were like, well, we need to make sure we have like Uh, a bigger venues. We ended up doing a call for the venue Way beforehand, but we're trying to plan it like way in advance Um, because even even with all that planning people still ended up going to the to the older venue or to the original one Be like, hey, I'm interact space. Where is everybody kind of thing? Uh, that maybe that's an austin specific problem And then I think I think I'm out of time, but I'd love to chat with you and give you some more thoughts on that too But thank you all for for being here. I really appreciate your time Thank you guys for joining the mentoring track. Um, our next talk in this room is at 130 Also, note that the expo closes at two today. So if you wanted to get to a booth make sure you do it before two Thank you. See you at 130 Can you actually hear a difference? Yeah, and the slides are actually on there No, I'm fine. All right. Thank you. Uh, yeah, it's all done Yeah, well Now we can start All right, uh, thank you for coming. Um So I wanted to cover a few key aspects of the fairly long history of the exam project. It's you know, 15 and a bit years old Um and Basically, I'm going to pick out a few sort of sets of stories and then at the end of it we're uh, going to look at why the lessons which Could be of interest for other open source projects and also for us in general And that story really covers, you know, very different aspects of the history of the project And some of them are actually not very well very well known so Before you ask the presentation is already at that, um, uh, link And I'll you can download it from there So My name is last curts. I'm the community manager of the exam project I'm also chairman of the exam project advisory board. That's a number of vendors. Um Supporting the project fund financially and I happen to work for citrix I do like flowers. So, you know, I always put up pictures with me and flowers on it. So this is actually a queen of the night, which is quite lovely and Never mind, and then I wanted to thank our country the contributors to their talk Andrew Cooper, Christopher Clark Um, me hide onto and george dunnab So I've been with the project since 2011 And so I had to pull in some of the longer term committers to get And concrete us to the project to get some of the bit some of the earlier history Of it all so What is xen? And xen project. Well, it's a versatile virtualization platform It was always designed to be a component In a software stack. So, you know making an easy to use end user experience was never really a design goal Which probably was a mistake? Well, you know If you look at kvm, for example, it's like a nice and easy to use out of the box and In xen's case that isn't really the case but That has enabled other things which will cover during your presentation So the way how I kind of often look at it is The xen hypervisor is a bit like an engine engine taking a car analogy and system integrators basically build a car with xen as an engine in it and You know kubes os is an example of that xen server and so on and so forth And then the project We have this sub project structure where we develop Several technologies which are related to xen. So there's the hypervisor itself. There's a project to do pv drivers We had a number of projects for specific goals Which were temporary like one to bring arm support to the platform or get xen support upstreamed into linux And there's a number of unicolonal related projects as well but To really cover the talk and and some of the lessons we have to go back to the beginning To 2003 when it all started and I wanted to look at some of the original use cases for which xen originally was designed And you know the primary use course was really server consolidation Um Then you know specifically also in the original paper They listed co-located hosting for facilities and distributed web services, which eventually became cloud computing And then you know a whole blurb around secure computing platforms and mobility and stuff like that But actually in reality, that's very closely tied to server consolidation with high availability for torrents and those kind of things What's really interesting also is that some of the terms, you know, like cloud computing and xen kind of historically go very well together Um, and if you're interested you can still find the original paper The original design goals are also really interesting. So There were like really three design goals. One was around modular architecture and One of the ways to implement that was desegregation Um We also have a very flexible architecture by user system services And a high degree of being able to customize the platform because The initial goal really was to target servers, but maybe also desktops as well And many of those are still relevant today And have enabled a very Interesting and complex ecosystem and i'm going to try and sort of explain that through a little bit of a idea and product Genealogy So we started off with server virtualization and on the side, you always see the kind of products Which are relevant in this space then In 2006 amazon and it was also slice host at the time which was bought by rackspace started to use xen in cloud computing and That really kicked off very widely What was interesting and i'm going to cover this a little bit because it's not very well known the Department of defense kind of took xen and forked it and built a whole series of platforms around that and there's some really interesting stories there um Then there was an attempt around 2012 2013 to bring xen to desktops There were a number of companies trying to including citrix to do that, but ultimately that flopped And but it was still an interesting milestone for xen because it enabled other things We Had a number of unikernels. So the first of those unikernels is Was galore. What was halvm? Which is a unikernel targeting haskell Which has been developed by galore um And then what's really interesting out of this whole desktop world We got specialized We i call it defense applications so that like the cubes are was very very and for dod for the dod primarily Targeting desktops laptops as well as tablets um Then that led to a number of vendors which are active in a space looking at expanding their use cases. So um, so we've actually seen platforms such as virtuosity created by company like dener works which use xen For mixed criticality applications in a different defense context and more recently and this is This happened entirely Surprisingly to me We have seen a lot of momentum in the embedded and automotive space so If you look at this right So how did we And there's also a set of technologies called virtual machine introspection Which can't work with the others. Um, which is a security technology So if you look at this How did we actually get? To such a diverse ecosystem. How does this happen because you know initially We were targeting server virtualization, right and what were What were the factors in a community which kind of enabled this and then the other question is of course, you know Is this a good or a bad thing, you know, and what would you have to do if you wanted to replicate this as a project? um so the first reason Why what we kind of went into this direction is That xen came out of a university project, right a lot of the Committers and maintainers and developers all Had a research background You know when when it was commercialized all the guys working on this stuff working at the university just You know became employees of xen source at the time So there was already a very strong Affinity Generally to research and cool ideas and you know just letting things which may not necessarily be You know good for the vendor behind the project happen and to support that covertly or you know directly um But also more importantly it also comes back to a strong security mindset Which you know which was already um Encoded if you want so in the in the original design goals So that kind of laid the substrate to make some of this happen So if you look at it, you know like so we looked at some of the design goals um So disaggregation for example was part of the original xen conception And that whole idea was influenced by microkernel thinking Um So let me just explain you know what I mean So if you look at a typical xen architecture, you have your hypervisor down there You have a virtual machine and then you have your dom zero and then we have a couple of boxes here So we have system services. So we were talking about design goals earlier so system services are basically Applications which run in your operating system and some of them can be replaced. So we have for example Two xen store variants. You can you notice one or you can run xen with with q mu as a device emulator or without one Today and there's like three different implementations of two stacks, right? Then on similarly You can replace the kernel. So you you basically run either linux or bsd But just for example work going on today to run xen without the dom zero at all Or to run it with an autos All kind of signs of that level of flexibility But coming back to desegregation desegregation is that idea To basically be be able to run some of the things you're running in your in your dom zero in some You know other other virtual machine and you decompose your system that way And that creates additional security boundaries and there were quite strong design constraints around that to make this happen. So all the PV protocols which are basically the interfaces between the various xen components were specifically designed for that use case and we had to sacrifice Performance and make other trade-offs to kind of keep this, you know, keep this going and this has been a constant throughout the life of the project um And then of course, you know, since the beginning security technologies were always in the mind of our maintainers and quite often, um, you know features which were not necessarily core To the project were just upstream without resistance Um, and sometimes with a lot of support from from maintainers so If you actually look at this um Disaggregation is a really interesting example because if you look at put it back onto the project genealogy You will find that in server virtualization and cloud computing That stuff isn't actually used, right? But it's used in all these other um products and markets and um If you then start thinking about community dynamics, right? That creates a degree of tension And that's not always that's not a very happy story to some degree You know tension is always like the precursor to conflict. So if you look at this um Until very recently all our key members of the project were employed by companies who developed products for the server virtualization market segment, right? and you know That's kind of the core use case and Everything else is kind of non core and in other words really that means that You know, everyone else who built products on the non core stuff really relied on the goodwill Of the maintainers to basically, you know sustain these other ecosystems and use cases And you know, if you look at that that's kind of You know, you kind of can see that sometimes they might might create some frictions and difficulty What's also interesting is that you know, we're now seeing a shift where um, where the community is becoming more balanced and that's Just started very recently. Um, and it's really accelerating um And that actually required a lot of actual work to make that happen So let's look at this tension again, right? So and and bring us back to desegregation because that's kind of a good example So First of all, we also have to remember that 10 12 years ago Well, yeah when when cloud computing kind of started the The attitude to security in general Um in our industry wasn't actually really very good compared to today, right? So I guess if you look at it in terms of analogies um You know, then the red security where we're now with privacy to some degree, right? um So what what happened so we had two things we had, you know, core maintainers, you know Not really having any, you know Not using a desegregation feature in there in the products which the employers, you know, funding them to work on um But at the same time you're getting features in And you have to kind of make sure that You know, this desegregation and the basic principles don't break but At the end of the day, you know, if you're working on a product, you're going to cut corners, right? So, you know, whenever whenever the core team chose Uh, you know, I had to implement a new feature, you know, they would Use the quick route or the more performance route for the main use case And that meant that, you know, sometimes those new features went into dom zero first Um Then of course also, you know, if you're a maintainer and you're familiar with your use cases and someone comes in with a very different use case um to really Understand that entire use case very well and then, you know, try and come up with the best possible approach That takes a lot of mental cycles, you know, which they're again often not prepared to do And so again, you know That sort of rigorous review which would happen for the core use case that hasn't always happened in those cases and You know, and that led to Consequences, so for example, you know Like a fairly good support for desegregation in a platform But, you know, nobody ever thought about adding build support, you know to it and the result then was that pretty much everyone on that um Graph is all their products who uses desegregation develop their own their own solution for building those things And you know that sort of duplication and increases the barrier to adoption For newcomers at the end of the day and it also creates unnecessary complexity So to summarize this section um So we basically the strong security mindset really enabled contributions which Were related to security Which at the time were seen as sort of non-core because you know, everybody talked about security, but really You know at that time It wasn't as important as as as features And there's plenty of this is really Quite unusual, you know, there's loads of examples of open source projects, which if they have a certain goal for their project If something which isn't quite fitting into this comes in There would wouldn't necessarily necessarily allow that feature to be upstreamed and We've been suffering from this ourselves Occasionally so like an example for example is is qmu And you know the primary use case is as a machine emulator or a virtualizer But xen itself actually doesn't need the virtualization support. It just needs the Device the back ends and emulated devices so xen kind of uses qmu like a A library for back ends and emulation devices And all the other stuff we actually don't need and we've tried multiple times over a period of 10 years to work with that community to have an officially supported configuration and we never succeeded And you know similarly You know another similar example is I think intel Try to do something similar for a slightly different use case And that then ultimately led to a fork called ne mu, which is no qmu, right? and so So that's just an example that you know, we were kind of somewhat unusual in that in that sense So let's get to the next story and That um leads us from you know From the classical server with cloud space to military clouds and desktop and tablet use cases And there's some really interesting nuggets in there, which I only found out about last year So that story started in 2007 at xen summit in new york And we're specifically looking at that fork from server virtualization to defense application in terms of product So at that event Do the employees presented new research around how would you be able to Um apply five common criteria security clearance for xen and a lot of this was about just Code review, you know, I'm really sifting through the code and And really checking that everything works as it should do And so there's obviously a correlation between the amount of code And the amount of time and effort it takes to do that And so you would actually want to take, you know, build a more minimal version to make this more Um financially feasible and I also developed the xen security modules That research then led to the xenon separation virtual machine monitor family, which today is widely used In all across the dod. I only found it out last year Um, when um when I was invited to participate in a security event, you know, which Those guys organize called the platform security summit if you google that you can find Some of the talks which represented that they're quite interested Uh, interesting subsequently, uh, the xen security modules were upstreamed But all that other work, you know, was just basically used by those military types and I never really tried Very actively to engage with the project or even upstream it So let's look at this in a little bit more detail because I I mentioned this already so the uh, so that research at the um, uh, at that xen summit in 2007 led to a prototype and uh, um And there were really two interesting things to highlight here. So, you know, they fundamentally Had a team of three engineers over a period of a few years kind of tracked the code base and you know, take stuff out and uh, and do the Do the actual certification to see how much effort it would be to keep on top of that and uh, and it really showed that actually the The model of design to some, you know, was validated through that activity and it also showed that you could create and maintain A cut down variant of xen was about three man years You know over Over two years. So it's one and a half, you know, that's quite a that's that's not too bad from a cost perspective if you really care about this stuff so That was kind of the one ingredient the other ingredient was an attempt by three vendors to bring xen To desktops And it was basically, you know, basically the idea would be you'd have your hypervisor on it Um, you know, you would have be able to run multiple, you know windows linux, you know, whatever you know all on your desktop and You know all those vendors thought this would be the next really cool thing But you know, it never really happened. It was a little bit like the year of the linux desktop, right or or other things like Yeah dual symphones and all these other sort of more complex things which never really picked up However this really Was an interesting stepping stone to cubes os it's a very active project and And then it also led to something we call open xt and secure view Which is basically the The dod equivalent of cubes os Which they basically use in various agencies So So that's kind of what happened And I wanted to really think about Why this happened and what enabled it So What the engagement of Some of those security types, you know with the project really did is it created a virtual cycle which Which made send a very attractive platform for security research and it's still the case today And basically, you know the enabler there again was we accepted those contributions and of course the The modular architecture and which we talked about earlier um Then the attempts to commercialize xenon desktops um Really late the foundation for a new class of project of products But you know again, you know, none of these capabilities were used in in in in the core use cases There's a similar related story um, which Played out very similar which was around gpu pass through and graphics virtualization And actually what's not very well known is all this stuff was actually developed For those desktop use cases Um, but then, you know, it actually turned out to this is quite cool stuff Which you could use in the cloud and uh, and so that's kind of an example of um, one of those no no Non-causing is actually been being beneficial for Um In the original use case cases So the next set of story is around virtual machine introspection so Let me just briefly explain what virtual machine introspection is. So if you If you're trying to find, um, you know, if you're trying to to to monitor your operating system for Intrusions like through viruses or the malware normally what you do, you know You install a virus checker or something or some software which you know monitors what's happening within the operating system You know for specific Signatures and patterns, you know, and if you detect it you quarantine it and so on and so forth um Virtual machine introspection because you actually have the hypervisor It's basically so so the the early approach is essentially equivalent to having Having a camera in a prison cell, you know where the prison cell is your vm um Where virtual machine introspection allows you to do this from the outside Which is basically the equivalent of a panopticon So it's basically, you know, like you have all your prison cells open and then you have your you know, like a A big tower which just monitors everything and That also started off in around 2007 at a at georgia tech Where A number of researchers, you know came up with this idea and started by research around it and in brian pain Created a project called center xs which later, you know became lib vmi, which is still around today Then in 2009 That functionality was basically integrated into into the hypervisor And again, you know This was like a non core use case at you know at the time. This was an interesting research project Um, um, you know, why would you want to take that into a platform? Which already had so many products built on top of it But you know to maintain a shepherd this through at the time And actually provided quite a bit of active help um Then many years later um Inter launched it has well and broad world cpu generation which had some features in it Which carried the promise, you know to To really speed up this whole concept of virtual machine You know if you if you if you look at how this works, you know, you're you're monitoring memory accesses, you know from the outside in a vm and you know The performance overhead of that is potentially quite quite high um And so those new hardware capabilities um Sparked interest by a number of security vendors to really look at this technology when I actually looked at it realized that In fact, the overhead really wasn't as big as they originally thought and I could already productize it um But you know as a result of that initial engagement, you know, they re-architected the subsystem, you know and put a lot of effort into it and In parallel, you know developed a number of companion technologies for it And then you know just two years later. It was productized And you know today a number of similar products available from a company called ais and a startup called scientific And there's ongoing contributions You know to those features, um, which indicates that some of those technologies um Are going to be included into further or existing new products So looking at this again um From a from a reflective viewpoint, you know, if you really take a step back actually Some of those security made features made it into the original Server and cloud use case so some of them were specifically developed for it like life patching That kind of happened as a need as a response to the uh cloud reboots. We had a few years ago But you know Most of these other ones just happened sort of happened by chance because somebody had a cool idea and then it enabled an entirely new ecosystem around it so I wanted to look at this in a slightly different way so so why What were the reasons for some of those technologies not actually making it into Into the core product for the core use cases in the in the platform. Well, You know one example is the the hardware capabilities to actually do this in an effective way when went there Or it was too early So there's a whole set of technologies Incent around Trusted boot and stuff like that which are used in cubes and a few other projects But they never really made it Into cloud computing until you know until now. So there's a lot of activity around this today primarily because of microsoft and Manating that for for newer versions, you know of windows going forward Also in some cases, you know, I talked about the extent security modules beforehand There use is complex and very use case dependent. So you couldn't really make You know make this easily usable in a general you know in a general sense and You know, it's also quite a lot of specialist Expertise was was required So that's kind of the one so the last part of our story is It's really going from From this desktop Use case which really didn't you know, which ultimately wasn't successful except for in niche in use to mobile use cases and embedded automotive use cases and that's where it becomes really really interesting and that's also where In the last year or two I've seen most innovation and disruption in the project is coming from vendors in that in that space And that story Started in 2008 in Seoul in Korea Where samsung basically released What I presented a you know a port of xen to arm um That was using Paravirtualization because at that time armcpues didn't have Didn't have virtualization support and both Me and Ian Pratt Well, actually was Ian Pratt at the time and later me were trying to convince samsung to really upstream this stuff But they never really we never we didn't we didn't succeed, you know samsung didn't want upstream the work Primarily, you know, I think at that time, you know, particularly the far east, you know, china Korea and all those countries and vendors in those countries really didn't know how to work with open source projects and At that time, you know, that was a big issue and You know in the last 10 years or so this has obviously changed a lot Um So we then settled, you know, we then basically said well, we can't get them to upstream this but we don't want You know, we want to draw them somewhat into the community So we allowed them to host, you know, their own separate for folk fork within the community and engage with them on an ongoing basis um Another meant that samsung, you know regularly showed what they did at at developer summits um The last thing they did was Uh A dual mobile operating system running, you know, showing that on a samsung On the very samsung devices and later nexus 10s Um, really showing, you know, like I had this cool demo where in one vm, you know, you had Angry bird running and on the other one, you know racing game and they've dealt with all the graphics virtualization stuff And it was really really cool At the time And really to do all this they had to do they had to implement real-time capabilities and they also Had to take some of the gpu work Which was done for the desktop case and bring that, you know, to the um Bring that to the arm environment So what this really did is You know, even so that was never upstreamed It showed people in the community what was possible And so it inspired others to really go and look, you know At this along Similar lines, right and so so that work from samsung fundamentally laid the groundwork for For the developments we're seeing now in embedded as well as automotive Where all these graphics and rich io capabilities As much needed as, you know, there would have been on a mobile device For mobiles that never really picked off, you know, you can imagine The user interface implications of having different personalities on your phone and so on that's all quite awkward I don't know whether you ever tried to use one of those dual sim phones Um, you know, it's a similar similar kind of problem not not enough market appeal for that But from my perspective, you know, I'm not actually seeing this as a failure And I was really a useful lesson and as a project we could really build on this So what then happened in 2012? um At that time arm brought out, you know added virtualization extensions to their cpus and That led to a complete reboot of the xenon arm port. So a number of people got together And started upstreaming xenon arm support, you know, properly using the using the Port that samsung had originally developed as a as a baseline and you know really looking at it and see oh They've done it this way. Maybe we can improve that right as a as a nicer and better iteration and learning from what had been there before and Interestingly enough, you know, so arm added virtualization extensions to To set to their servers to their cpus to target servers So, you know at that time everybody, you know assumed well in three or four years, you know arm servers would be really Quite successful and you know create a really tough time for intel, but you know, that hasn't really happened And of course, you know, we really learned from that early arm port. We ended up with a clean and simple architecture Um, there was a really great match between the arm instruction set and xen's architecture And we ended up with a really tiny hypervisor So the initial port for one platform was around 30 000 lines of code the equivalent of x80 of the x86, you know code would be around 200 000 lines of code, right? and The fact that we had that upstream support and the simplicity and smallness of that, you know of that arm port really suddenly sparked interest by By embedded vendors and for embedded use cases But there's a missing evolutionary link here because you know, we were talking about um vendors who were active in the sort of military space Having an interest in xen and that happened before we did the arm port, right? so there's a company in in called dirner works And they launched a product called arlx which initially was an x86 product they added real-time support to its scheduler support and and then later safety certified this thing that this product um for Avionics standards, right? and so that that sort of showed that you can actually take Xen take stuff out and then get it safety certified and uh, you know Have it run in drones or you know offer some software on planes and similar things and That then again, you know created that bridge, um to uh To the next evolutionary wave, which is really around embedded in automotive So this really started picking up in the last few years We had our first automotive based platform in 2015 Um, we had a first embedded arm only xen distro developed by xilinx in 2015 and in the last few years there's been an incredible amount of rnd and a lot of You know, I'm really excited about the next one or two years because this this entire space has You know create so much innovation and new ideas and it's really it's really challenging What's happening within the community? there's another little thing I wanted to talk about And this is so I I thought I called it sedimentation because also what we've seen through the lifetime of xen is that Stuff that used to be implemented in software Eventually gets you know moved down to the hardware And I thought that'd be a really a few really interesting stories around this um You know an example of that is we used to do interrupts and timers fully in software And you know today on newer hardware, we just used the hardware support and we kind of constantly had to adapt to that and there's some It's also really interesting those silicon vendors like intel and amd, you know, they have always really worked very closely with xen and so The way how these instructions sets and and this hardware support has evolved is really heavily influenced You know by that butted interaction and I thought There's some really interesting stories. We could tell but actually as it turns out I can't really tell these stories because the the These cpu vendors hadn't actually engaged with the community But they had engaged with vendors in the community and so all of this happened under under nda's So to close so What have we actually learned within the community and you know, what's there to learn you know for for you guys and The first big thing and that was a really big scenes as things through throughout that presentation Is that culture really enables innovation, right? So So because we had this culture to just accept stuff Into the code base, which wasn't core This really you know, this really enabled things interesting directions, which nobody thought could happen This wasn't really planned in our case We just inherited this due to the you know due to the project being a coming out of a university But you know, we didn't really do this well at the beginning, you know, we Sacrificed a lot of due diligence, you know initially and we could have done that, you know better And we have done it better in the last few years We also now But also didn't do this at the beginning really help companies and organizations to You know, if they come to us as a you know and say well, we want to do that We just won't shut the door. We will actively work with them and try and find the best approach And also from a community perspective, you know, we always have worked with new ideas And with forks of the project as well as at distributions so examples there are At our developer events, you know, we always gave airtime to new ideas So, you know, we're you know in the early days of unicunnels, you know I always had a platform within the community. The same is true for many of the other things Forks aren't always bad You know, you think oh my god a fork is happening, you know, we just It's a bad thing, but it's actually not Not necessarily always a bad thing and if you If you have downstreams With special needs, you know, you really should work closely together with them And that's also something we haven't done very well at the beginning But we are doing this, you know a lot better today Another thing really is there's a big I didn't talk about this in a lot of detail, but if you look at this You're the way how you do your process Can either enable or hamper, you know innovation so We had to sing incense straight from the beginning where we have we have like a feature life cycle You start off with something experimental and then something, you know And then we come tech preview and then eventually we become supported and You know at the beginning we kind of use this life cycle as a back door to get You know good to get stuff into the platform, but actually also market As maybe you know, maybe you need to polish this a little bit, you know before you actually build on it Whenever there were larger contributions of features which weren't You know which weren't part of the non-core use case who would always insist that the contributor would step up as Because we didn't want to have stuff in a platform which would spit rot and then, you know, obviously um Maintainership itself isn't you know, isn't enough they could go away. So we always have to have From the beginning, you know the capability to deprecate and remove features, you know If it actually looked as if they you know as if they were becoming stale and we have Exit, you know done this quite a few times the other thing is Is that the tension within the community you will have this tension if you have You know if you have people and companies which come from very different backgrounds and directions there's gonna be tension and these can lead to conflicts and We had quite a lot of arguments between those groups and you know, they never really go away and they can go get quite ugly and that really means You have to do, you know active community management. You have to you know, you can't step into those Conversations and try and shut them down right from the beginning because sometimes Creative tension is needed to lead to something interesting. So that requires quite a little bit of skill and it you know, it's also a little bit risky sometimes Another thing which was really surprising to me is that if you introduce a new process that can sometimes lead to a significant increase of tension so around 2013 2014 and you know again about two years ago We looked at our security vulnerability process and we kind of formalized it and tightened it and so on and that had unintended consequences because you know, like All our core developers were focused on server and cloud computing And you know, and we have the security team, you know, which is security response team, which had all security vulnerabilities Primarily, you know for those two use cases and But what happens if you find a security vulnerability in code, you know For something like cubesos or something, which you know, isn't in this cloud development segment and you know, when we started to formalize this Suddenly it raised the question. Well, you know, who actually has to fix this, you know Who has to jump drop everything, you know and And fix an issue in that in that piece of code Which some other company, you know, which i'm not being paid for Is actually benefiting from or some other project And that led to a lot of Tension and then in the end, you know, we had to resolve it In this specific case, you know, we just adapted, you know, our process and basically said well You know, we can have supported features Without security support or You know, we basically say well, you know, if you have a major feature and you have a product built on top of it And you wanted security support it Nominate some contact point, you know, and then if an issue a person is responsible for that within your project Or your organization and we, you know, if that issue arises We're going to come to you and you know, you fix it and work with us And We also started to heavily use kconfig to be able to build different Configurations and larger features which, you know, which are non-core are basically build or runtime disabled And we use sub projects as incubators for for new ideas And these are really seeing things, you know We have an automotive project and in this in that area We're currently looking at things like how do we deal with safety certification better and so on and so forth And that's basically it. So if you have questions, please go ahead No questions. That's all right Don't worry about it. Thank you very much Thank you for coming Okay Working yes, maybe okay good Now I'll turn it back off Seems like as good a time as I need to get started since it's three o'clock It's sunday the end of the day almost the end of the day. There's one more talk slot after So not the very last So but I do appreciate everyone sticking around today. It's a scale to Not not for the faint of heart to make it all the way through. So awesome Awesome and for like probably the best weather has been today, right? So, you know, there you go So thank you for coming to my talk. This is called fight flight or freeze releasing organizational trauma A little background because I don't have an interest slide here Because quarry quinn told me that resume slides are bad and I should feel bad if I have one So I took it out of this one, but my name is maddie stratton I am a devops advocate at pager duty And that is probably one of the last times you're going to hear me talk about pager duty throughout this entire talk If you want to know more about pager duty, we can certainly talk later As far as things like that go And I I really really love this talk. I'm really excited to be able to give it here I do have to you know, since mary is in the audience I'm going to give a little shout out So the first time I gave this talk was at a conference Over the summer called redeploy, which is a conference about resiliency engineering and A little bit of the background on this is I had was talking to Mary's co-organizer jpaul read about Some ideas as I was learning about post-traumatic stress and I started to say like oh, this seems really similar To some organizational issues and he said you should totally write a talk about that and we have a cfp open And I said well, I'll submit it mostly because I want to get Feedback from you and mary on the submission and then they accepted it and I went Shit Now I have to write this talk Uh, but it was uh, it was a really great journey to write this talk and I'm glad to be able to share it with you So first of all just a little bit of a content warning. So we'll be talking about There's some discussion of trauma and post-traumatic stress. There's no specific I don't talk about my specific trauma or anything like that, but Be aware that we are talking about post-traumatic stress a little bit Disclaimer the second so while I am a trauma survivor I am not a mental health professional. So none of this is Should be taken as you know direct medical advice or anything like that But these are some analogies and experiences from things that I learned Being a survivor of post-traumatic stress So with that, let's get into the talk So consider the zebra So if we have a zebra that's at rest, there's no threat coming on. He's just zebra's just chilling around being a zebra self Right, so this is operating the zebra is operating within what we call rest and digest This is the rest and digest functions of what's called the parasympathetic nervous system So there's no threat of predators. There's no stress As its name implies, you know zebras mostly think about eating and maybe about making more zebras depending upon You know what where what's going on with that zebra? Now We introduce a lion now. There's a predator. Okay, so now a lion is coming and chasing the zebra So there are drastic physiological changes That happen with the activation of the fight or flight response of the sympathetic nervous system So the zebra's heart rate increases its breathing increases its pupils dilate There's large amounts of stress hormones like cortisol and adrenaline are being released into the zebra's bloodstream Its blood pressure increases and any non-essential functions like digestion. They come to a stop The zebra's nervous system is preparing it to literally run for its life now Let's say the zebra gets caught. Sorry zebra I need to get you caught by a lion in order to make a point So say the zebra gets caught now the zebra's nervous system is overwhelmed. It has no more solutions So this is what's called the freeze response the zebra basically plays dead and it just freezes and it stops This is also a point that we might consider in a non zebra. We might consider this the inflection of trauma So Zebra gets caught by the lion plays dead lion drops the zebra kind of moves along maybe to go try to catch another zebra Now and so the zebra is maybe able to escape. What happens? What does the zebra do it actually literally shakes it off the zebra its body shakes And it's returning back to rest and digest. It's making a physiological change. So this is what happens When it comes to zebras now, I have if there's nothing else that you take away from this talk It's going to be this next point i'm about to make Which is that humans are not zebras So if there's a slide to take a picture of it's probably this one So all mammals have the autonomic Autonomic nervous system in common, right? So there that's the fight or flight Response this is this is common for all mammals Now as humans we have this thing called a prefrontal cortex Zebras don't have it lower mammals don't have it. It's something that's unique to humans And the prefrontal cortex is usually an advantage, right? This is uh, where executive function happens The prefrontal cortex is how we differentiate between good and bad between same and different How we understand consequences because we have this prefrontal cortex The disadvantage is this is where we mentally replay traumatic scenarios Which activates that sympathetic nervous system exactly like a real threat would here's the thing We all think we're humans and we're super smart. We got these super smart brains I'm here to tell you that your brain is actually kind of dumb, right? Our brains can't tell the difference between real or perceived or remembered trauma So when something happens that is similar to a traumatic experience that happened We have the same physiological responses Zebras don't have this right other mammals don't have this they don't go and replay this trauma the zebra that got caught by a lion It's not going back and every time it sees a lion is is going and having this same response Because of the one time it got caught So uh, so dr. Peter Levine has said so animals in the wild they're not traumatized by routine threats to their lives Humans on the other hand We're readily overwhelmed and we're subject to the traumatic symptoms of hyper arousal shutdown and dysregulation Is one of the few slides I will actually just read out loud to you. I don't usually do that But this is a really important quote. So peter levine has done a lot of research on this fight flight or freeze response And I'll have some links later to some some work that he's done So, uh, let's consider a healthy nervous system. This is what something that we call The window of tolerance. So arousal or activation occurs, right? Something happens Lion right So we're going to do that that triggers a sympathetic nervous system, which causes a physiological response And then we settle which brings us back to the parasympathetic and then something will happen that might Cause the response again We kind of bring up and as long as we're staying within this thing that's called that's what we might say the Normal range it's the window of tolerance, right? So the window of tolerance is a zone of emotional arousal that's optimal for well-being and executive functioning So this is normal, right? You're not you're not steady state all the time Something happens that requires a response. You need to respond the trigger The difference is can you kind of come back down to settle when this happens? Now what happens when we talk about an actual traumatic event? This is when we start to see so in an undisturbed traumatic stress you have a dysregulated Nervous system and I know I'm not supposed to move but I'm going to because I want to point So we kind of sat there. This is our normal window of tolerance Our normal thing we get triggered something happens and then something really traumatic happens and we can see how Our response becomes very dysregulated. It can become very Up down all the order and we can either either be continually flipping back and forth or we can potentially be stuck on Or we can become stuck off So symptoms of when we're stuck on is when we are traditional symptoms around anxiety or panic, right? The inability to relax emotional flooding sleeplessness a digestive problem which shouldn't come as any big surprise because this is the opposite of rest and digest So our body is physiologically taking resources away from digestion Or we can be stuck off which are some of the typical symptoms around depression. You have lethargy Exhaustion when you have chronic fatigue. These are symptoms of being stuck off a lot of other things and again Poor digestion because now you're spending all your time in rested digest You should be able to be going kind of back and forth Hyper arousal, which is stuck on And similar to fight or flight right, which is sort of that up here Hypo arousal is similar to the freeze response where you're kind of stuck frozen So we're going to come back to the window of tolerance when we talk about organizations in a little bit Here's the thing trauma is not Simple there's a lot of nuance to it and a lot of things to keep in mind Which is why it's not a simple scenario and that this is why this is not a 10 minute talk And it's a little bit of a longer talk because there's some nuances. We're going to talk about So here's a couple things to think about right so first of all, you know the nervous system has been overwhelmed So it occurs when a solution the active response to a threat doesn't work Right, that's what causes something to be traumatic because things happen to us all the time That aren't necessarily trauma. They're just necessarily maybe there's bad things bad things happen to good people But if we have solutions to them, they're not trauma trauma is when we don't have a solution when What our nervous system has been overwhelmed by something Again as you talked about the nervous system activation is the same for real or imagined threats Right, so it can be a real or perceived threat And then finally and this is really important. It's subjective and relative, right? So what I perceive is overwhelming or my capacity for a solution may be different than what you perceive as overwhelming And having gone through Work around post-traumatic stress and everything It's quite common to for me to feel like wait a minute What i've gone through is nothing compared to what some other people have gone through but again, it's all relative And different people have different capacities And different experiences How does this actually apply to an organization? And one of those kind of key things that all of those pieces happen Right, like when we talk about the nuance of it and again bear in mind that just like different people Respond differently and have different capacity different organizations based upon the humans that make them up depending upon the business that they're in Depending upon regulations all sorts of other things Will have different capacity for for dealing with this. So let's kind of go back To our deregulated nervous system and see how this might apply to an organization when a traumatic event occurs traumatic event being an unplanned outage Aren't all outages really unplanned I guess really an incident something That is beyond the norm Right and not all incidents are traumatic for an organization either some of them And we're going to talk a little bit how we can keep them From causing us to go into an organizational state Of undisturbed traumatic stress or into an organizational deregulated quote-unquote nervous system of our organization So we'll think about hypo hyper arousal, right? So that's the fight or flight So if an organization this is being stuck on right? So if your organization organizations that are hyper aroused They display efforts effects around Constant vigilance and sounds like they're fighting Voldemort, right? They're hyper aware of threats And and this takes energy away from moving forward. This is kind of constantly being on the defense Right. These are organizations that are stuck on A lot of times this is reflected in how leadership approaches outages and issues We hear a lot of wartime metaphors a lot of military military metaphors a lot of we are we're in wartime right now Another symptom of this is when you have production support teams when you have teams whose entire job Is to be watching production And supporting incidents and outages, but they're not but it's not actually connected to the innovation of the organization These organizations have a really hard time moving forward because they're spending all their time Looking over their shoulders, right and and being you you need to be aware I'm not saying that you should just say, ah, you know, I mean shit happens. Whatever, right? But there's a there's a line We can also become hypo aroused, right, which this is when we're stuck in freeze These are organizations that are afraid to make any kind of a change because to be quite honest What causes incidents usually a change, right? So we can in an organization that it becomes hypo aroused Again have trouble innovating and moving forward because it's sort of the sphere of if we change something Something's going to go terribly terribly wrong. So we're kind of in this down state all of the time So bear in mind that signal again remember when we think about How trauma comes in just like our brains can't really tell the difference between real and perceived threat as organizations we often Try to pattern match, right? We say this looks similar to the last time we had this massive Cassandra outage So therefore because we see a signal that looks kind of like that It's probably everything's probably about to go completely shit So let's mobilize everything even though the signal might not necessarily be indicating the same kind of a thing So we're looking for pattern recognition because that's something we want to do with humans and And while the signals may seem the same as previous examples of seeing that we've seen The complexity of the systems that we work with mean that things are never that simple right The days First of all bear in mind, right? So one of the things to keep in mind is that our systems are always in some state of degradation Something is always broken, right? And depending upon what little part over here is broken versus this part versus this part versus this part Means that while this signal may have looked the same last time Doesn't mean that this thing is broken at the same time that this was broken that this was broken Which has actually caused the incident So how many people flew on an airplane to get here? How many people have flown on an airplane in the last 10 years? Okay, right so fair enough. So let's think about taking off our shoes Right, so in december 22nd of 2001 cat named richard reed He tried to ignite explosives hidden in his shoes on a flight from paris to miami. It did not work It was a bad idea for lots of reasons. It was poorly executed. Thankfully But as a result the tsa began randomly searching people's shoes They're like, oh, well some idiot decided to try to like put a bomb in his shoes So therefore most shoes must have bombs in them. So we should probably check for them And then in 2006 the tsa mandated that all passengers must remove their shoes going through security And then you know a few years later said but if you give us some money, then you don't have to take off your shoes So the point being We saw a signal which was wearing shoes on an airplane Meant potentially destructive effect Therefore we're going to watch for all of them again and where this comes into it's kind of a phrase That someone you know said uh, so jennifer brea said, you know We have a saying in medicine that when you hear hoof beats the first thing that should come to mind is a horse Not a zebra and that this two cute by half phrases killed so many zebras So the this at its surface this sounds logical. It's like, okay, let's not look for edge cases Let's say this looks similar to the thing that happened before Inside of tech this is very similar to jumping on that first root cause This is sort of one of the the inherent problems with using the term root cause root cause implies there is one factor that caused the issue and the problem with that is It's rarely one thing and What happens here is when we find that first thing we see and we say that's our root cause Well, that's just thinking that everything was it was a it was a bunch of horses But maybe it was a zebra also. Hey look more zebras. So there's a theme So you want to identify your organization's window of tolerance? So what can we do to help kind of understand what that may be within our organization and how can we identify When we're becoming deregulated So I I put up a quote from dr. Peter Levine So I've got another quote from another very well regarded person in the industry Who has a lot of insight into that? me So my paraphrasing are kind of taking dr. Levine's points and applying it a little bit more To to tech is saying so a resilient organization. They're not traumatized by routine threats to their mission or business Um, I still have business spelled wrong in this slide. I gotta fix that So see and you notice this did not traumatize me. It was okay. I'm moving on Uh, non-resilient organizations are readily overwhelmed and subject to symptoms of overreaction shutdown and lack of Effort so that's many strength. I'm not a doctor. So what does this mean, right? So If your organization is resilient Doesn't mean that things don't go wrong It doesn't mean that you don't have incidents or outages But they don't cause trauma for your organization You're able to withstand them and that's a big kind of key thing about when we think about the idea of resiliency Right resiliency does not necessarily mean stability does not necessarily mean 25 9's it doesn't mean 100 uptime It means that when things not if but when things go wrong, you're able to respond in a practical regulated Normalization of effort and it doesn't cause everything to go parachaped right and so let's think about that So this so first we want to talk about regulation Not referencing war and g I might have a little bit of 90s stuck in my head because we saw Captain marvel last night so But how do humans and organizations take this deregulated state right? We talked about what it looks like to be deregulated How do we become regulated because it's one thing to understand that it's happening We talked about what some of those symptoms were we talked about that you might be hyper aroused or hypo aroused We're seeing that we don't want to move forward. We feel this stuff. That's great. You're saying great maddie We're we're not regulated, but I super want to become so what are so first of all we'll talk about how humans do it How we do it as individuals and then maybe how we can take these same metaphors and apply them to or an organization So one of the treatments for post-traumatic stress is something called eye movement desensitization and reprocessing also known as emdr How many people watch the netflix show russian doll recently? So there was actually a little emdr going on in russian doll. It was very cool. So there's uh the idea So patients difficult memories are offset with a positive association that's reinforced through external stimuli So that can be blinking lights. It can be little tappers in your hands It can be audio stimulus and so there was some of that in russian doll That's not really a spoiler if you haven't seen it, but Yeah, if you do watch russian doll and you see the part with kind of the lights You're like hey, that's that thing that maddie talked about so what's happening there is it's not Well, what we're doing is taking the the patient is going through kind of a safe journey But it's being reinforced by stimuli that's activating both sides of the brain Now we can't really do this For our organization We can't put giant strobe lights on either side of the office when things are happening and have somebody do a walk Through it doesn't quite work Literally the same But we can take the same principles which is taking Making a a positive association with memories of trauma Through some type of muscle memory stimuli And we can do that for our organizations. So here's a couple ways that we can go about doing this So one are the ideas of game days So what we're trying to do with this is we want to make an association of outages and issues with being safe With being okay So game days can help us do this. So a game day is when we're doing intentional Outages and we're exercising our practice of what happens when we have an incident what happens when a system goes down The thing is about game days they can be really helpful But they have to be done properly The idea of a game day is not to like kind of go psych out caught you Right because all we're doing there is creating extra stress for the participants We're trying to keep we want to keep them low stress and safe The point isn't to practice under pressure. It isn't to create a high stress environment So we get good practice on being under high stress. It's exactly the opposite It's to take this association of a thing that can be high stress and associate it with a physical Memory of low stress. We're associating response with a safe environment So when we're doing these explorations in a game day, there has to be guidance, right? This isn't just something we go due to our support team or to our SREs and to say we're just going to go break some Shit and yell at you till you fix it, right? We have to have guidance through this. It's a lot like meditation When we're when we're helping people who have post-traumatic stress through meditation So with some kinds of trauma meditation is really helpful But the problem is people can go into this inner what's called the inner landscape And if they aren't prepared and they're not guided Sooner or later, they're going to encounter The trauma right it's going to be there and Then what do they do? So they could be overwhelmed by it Or they find a way to go around it And similarly so if we don't have plans in place for our game day We're going to maybe get wound around the axle around the wrong bits of our process We're going to be too concerned about the physiological response. We have a stress We're going to start thinking that this is about training us to deal under pressure And the idea is not to make us better under pressure. The idea is to say that when these things happen, they're actually not pressure situations Also, we can in many and when we have meditation for trauma You can run into something that dr. Peter Levine called the bliss bypass Which is a way of avoiding the trauma which is the meditation is so peaceful and so calming That it just makes me kind of forget that the trauma existed, but it doesn't make it go away So if our game day is in following our usual rules of engagement around incidents, we've actually made it too safe Right, so we still need to follow our usual process and procedure And so if we're talking about things like plan failure injection, so if we're doing planned failure injection in a game day Run it like a real incident Here's the next time I talk about pager duty in my talk Not talking about pager duty the product but about how we do things at pager duty So we have these things we call failure fridays. So on failure friday, we will intentionally take a system down A system that we believe to be resilient And we follow our full incident command process This is the fact how we actually train our incident commanders before you go into incident command rotation and production You have to have run a failure friday So this is what's happening is by following even though the only difference between a real incident and failure friday Is that we actually go into failure friday knowing what is actually wrong? Because we actually broke it on purpose, right? But we practice This is creating an organizational association as incidents being a safe Normal place and in our case we actually run our failure fridays during business hours because It's actually the best time for us to Do it we don't have maintenance windows pager duty does not have a time when we can say well It's totally fine if people don't get alerted, you know from friday night from you know 8 to 10 p.m So that being said for us the best time to run failure injection is actually during business hours when we have everybody Already in the office on site. We know what we're doing This may be different for your organization You may do planned failure injection if you do and you may not necessarily do it in in production You can also do planned failure injection in a pre-production environment, but however you're doing it Run it like you're running a real incident This is another reason why you should run all incidents at their initial severity, right? So let's say An incident starts out and it comes in as a sub one You may decide later that this wasn't really a sub we may say, you know what this didn't actually Require our major incident response. It was a small thing Run it as if it was a sub one all the way through and the reason for this Is because it gives first of all it does two good things One that has nothing to do with the trauma stuff But I feel like preaching about it is this keeps us from litigating severity during an incident Which is really really important. It keeps us from saying we're going to spend time when we should be restoring service Arguing about whether or not this was a sub one or a sub two The other reason why it's good to do this is we don't want Sev ones to be is it going to sound controversial? We don't want them to be considered a bad thing They're just a thing that happens And if we run it as a sub one, it doesn't mean that we did anything wrong It doesn't mean that we're a bad organization. It doesn't mean that our systems are bad And all that happens is if we had an incident come in that we thought was a high severity to became a lower severity The worst thing that happened is we got to have good practice of our incident response process And it just went back to making us feel like this is business as usual. This is just the thing that we do It's no big deal Now we have to process Failure and this is really really key because post-traumatic stress happens when you have undischarged trauma When you have unprocessed trauma when you haven't been able to work through it So when we talk about processing failure and we're thinking about things like after incident reports and post mortems The ideas around blamelessness are really just the beginning Excuse me We need to process the failure or the outage throughout all the information that we have And and this is key the processing has to have a conclusion. It has to end Otherwise it's unprocessed trauma. We didn't finish So there's this misconception that in order to process trauma All you have to do is just get it out, right? If I can just talk about this enough Then I will be done with my trauma So that's not quite how it works. We need to integrate our experiences into a more coherent whole This includes telling our stories As well as changing our autonomic nervous systems Which in the case of an organization has to do with how we respond to outages and incidents So storytelling Is really really important We have to tell our stories Because that's how we process them That's happened for individuals. So this is part of Therapy for pts has to do with actually processing it But the larger organization needs to be able to tell this I did an episode of a podcast called greater than code and we talked about this particular topic And got into kind of this some some ideas around how Postmortems could really become a storytelling exercise As opposed to just a fill out a form because the form is sitting there in pager duty that we have to fill out And tick the boxes and say yep, I sure as heck did fill out the postmortem form You have to be able to tell the story and they need to become interactive A right only postmortem does not help anybody It has to be experienced So I'm going to talk a little bit about something called somatic experiencing and soma means body All right, so in a classic top-down therapy We start with the emotions Right, why do you feel the way that you do? This is very similar to an enduring it with an incident or an outage looking for root cause first Saying that's the first thing we're going to do is try to figure out what the root causes And the reality is that's kind of one of the last things you want to do is think about contributing factors because we're trying to restore service So in somatic experiencing the body is worked with first This enables a titrated approach to the emotions So we aren't necessarily saying we're going to think about all the emotions all at once We're going to think about the body's response first and that's going to surface Some of the understanding of where the trauma comes from and the letting us process it So similarly during an incident you are first working to restore service before we worry about And it's really important to look at the whole system When we're processing it not just starting with the quote-unquote surface emotions Which are the symptoms of the outage again when we talk back when we think about the symptoms of the outage Those are those signals that may point us down the wrong road that may make us think this is a more traumatic experience than it was Because small incidents and large incidents sometimes have very similar symptoms and signals You're going to spend a little time talking about these ideas of cognitive distortion So cognitive distortions are exactly what their name implies their distortions in our cognition So these are perspectives with bias They're irrational thoughts and beliefs that we unknowingly reinforce over time And some of these can be really really prevalent in how as an organization as a team as a squad how we approach incidents and outages So these patterns and systems of thought are often really subtle It's really difficult to recognize them when they are a regular feature of your day to day thoughts So there's as many as uh 16 generally accepted cognitive distortions I am not going to go through all 16 of them If you feel like looking them up google is a thing We're just going to touch on a few that might be specifically causing issues within your organization So polarized thinking This is also sometimes known as all or nothing Thinking this is seeing everything in extremes, right? So this is it's either perfection or total failure, right? So remember that our systems are always in some state Of failure. We don't have perfection Similarly things are not always completely asked Right something's probably working at any given time So we need to not look at these generalizations of it's either everything is beautiful and amazing or everything is shit Because what will happen is this this particular distortion Is going to affect your ability as an organization in two ways if you're either thinking about All you're going to be chasing a goal that you're never going to hit And you're going to think that you're not succeeding because You are not in this blissful state of a hundred percent uptime Or on the other side if you're thinking that everything is terrible everything things can feel very uh disinheartening very much like things are just awful and why should I even bother And over generalization This is where we take a single instance and we generalize it to be an overall pattern On a personal level. This could be something like well. I got a C on this test. So i'm stupid and i'm a failure Right in an organization. This can be well. We had a SEV one incident on this particular service So this service is unstable and awful Um when we combine this with another distortion called the mental filter This can manifest as seeing only the negative and eliminating all positives about a situation a person a team or a system Um fortune telling we do this a lot in it Um, we feel like if we have enough data We can be predictive. Hey machine learning, right? So if we only know enough We can predict the future Well, yes, we can start to get ideas on what is likely to happen But guess what? I'm sorry to say. I don't I don't care how much blockchain you throw at it You can't predict the future with certainty. So stop trying And here's the reason why our systems are you know, you're probably tired of hearing me say this our systems are complex We have to understand that our predictions are not Facts They're actually just one of several possible outcomes Does this mean you should not be looking at tools around ai ops and around machine learning and and how we can start to have some predictive analysis By no means these are all helpful. However, they are not Magical spells that will tell you what is exactly going to happen And then we have a control fallacy. So this manifests as one of two beliefs So one we have no control over our lives and we are helpless victims of fate or two We are in complete control of ourselves and our surroundings giving us the responsibility for the feelings and actions of those around us Both beliefs on either end of this spectrum are very damaging and both are equally inaccurate So no one is in complete control over everything that happens to them, right? And no one has absolutely no control over their situation So even in extreme situations when the individual seemingly has no choices in what they do or where they go or what they say There is a some Control over how you approach a situation mentally now as an organization within tech where this comes in is again If I could just keep those damn developers out of my system I could control everything, right? If we only had enough tests, right? It's like I can control all the things or on the other side That's not my fault everybody just f's everything up all the time And we have some somewhere in the middle So let's think about how we can make this better, right? So how many people in here Are currently part of some kind of on-call incident response rotation? And people have to deal with that. How many people have had this experience? Or potentially maybe manage teams or work with teams that are incident responders of some kind How many people work for a company where people give a shit if things break? Okay, right. So hopefully be everybody. Okay So we wanted I'm just going to kind of finish up with some thoughts around self-care around things that we can do to make as individuals On-call an incident response less traumatic So why does it matter? Why am I even talking about this? So burnout is a real thing and We at okay, I guess it's the third time. I'm going to mention pagerd. We We did a survey about a year ago across. I think it was about 12 to 18 thousand different incident responders to understand um Who had who had left their position in the last 18 months left their company? We wanted to understand why? and For the vast majority the most important metric related to attrition was not money I mean some of them left because they wanted to make more money most of them didn't It wasn't even their manager, which is generally a true thing I mean sure we've heard the saying people don't quit a company. They quit their manager The most important and the most relevant metric related to attrition Had to do with how often they were paged How many and it wasn't being paged at all. It wasn't like well I left because I want to go somewhere. I never ever get paged But it had to do with how many nights were interrupted how many workdays were interrupted how many weekends were interrupted so this really really matters and It can matter to you as an incident responder because you don't want to get burnt out because you actually care about yourself as a human It could care you could matter to you as A leader in your organization because you care about the people who work for you Or you may be a leader in an organization who doesn't really care about the people who work for you But understand that it costs on average $400,000 to replace an engineer And it's a lot cheaper to keep people than to have to replace them. So let's try to not burn them out So a couple things that we can do When it comes to self care around on-call and incident response So one is doing some context switching Create a ritual around going on and off of call This helps your body understand that things are changing. So, uh, for example, like when I take out my contacts, I get sleepy And this is because over the last worth 20 something years or whatever my Body has associated it with when matty takes out his contacts It means it's time to go to bed and likewise when I put my contacts in I get a little bit more awake because that's the thing I do in the morning So my body has some expectations So there's things that you can do if you say like when I go on call I physically make this shift and I uh Was in an open space I think it was at devops days charlotte and someone came up with the idea and said like Maybe you have an on-call hoodie This is the hoodie that I wear when I'm on call And then I take it off and I never wear it if I'm not on call And of course I went and like oh that would be some good swag Right could have the on-call hoodie You could have things that have to do with like when I go off call I treat myself to this one particular tea now the trick is whatever these rituals are They need to only happen when you're going on or off call because you're creating this physiological response You need to allocate mental bandwidth So if you're on call, that's really has to be your number one priority Don't be trying to deliver high priority projects at the same time that you are the level one first Responder for instance for your organizations Having a good relationship with your manager is really key to this You have to work with her to help understand where you're coming from With this and help set expectations, which might say hey, I'm going to be on call this week Which doesn't mean I'm not doing any other work, but There's only so much mental bandwidth I have so I need to be able to shift and maybe I'm going to have more of a focus on Resolution and remediation type projects during this time than innovation type projects because I'm context switching treat yourself Have an awareness That on call is a stressful activity no matter how much amazing stuff you might do With your organizational EMDR and all the things I've talked about over the last 40 minutes It's always going to be stressful. I went I you know, I spent 20 years and various forms of system administration and Technical operations and most of those two decades I was on call almost all the time and then I went and worked for a software vendor for a few years and Blitzfully hung up my pager and then I joined pager v. Okay four times pager duty and now I am part of an on-call rotation and It was really stressful the first time Every time I go into my one little 24 rotation every two weeks of being innocent commander Just being on call is stressful to me so Knowing this knowing that on call is a stressful activity take some extra time to treat yourself during these times to counter that stress So I talked to Anna Medina from gremlin And she had some good suggestions and one that I really liked she said the weeks when she's on call She schedules wine or dinner with her girlfriends or a plan to kind of go out for her favorite dinner Because she knows that it's going to be a really rough week But what happens is it actually gives her something to look forward to so this is my on-call week And I know it means I'm going to go have my wine time now again It depends upon you know Maybe you can't go out and have wine with your friends while you're on call because maybe Your expectation is you can be on a terminal within five minutes But find something you can do or even if that's the case if you say hey I'm going to be on call for a week during one of those nights I'm going to take it I'm going to take a couple hours and go out with my friend Sonia And you know what I'm going to go to someone say can you take an override? Can you cover me for those two hours because I have to go do this thing Stay healthy So it's very tempting and common to fill ourselves with sugar and caffeine while we're on call because We're probably going to be up in the middle of the night, you know And it's sort of the traditional way we think about responders and incidents is like Drink a lot of mountain dew and eat a lot of ding-dongs and twinkies It's kind of detrimental to eat all this junk and expect our brains to still be able to be Performance so eat healthy stay hydrated and drink a lot of water There's a physical effort to being on call It's like being an elite athlete In a way, maybe I should elite athlete keep healthy. So take care of your body and that will take care of your brain And it wouldn't be a talk for me if I didn't just crowdsource to twitter to fill in the slides that I still needed to do So as always I went to twitter to get some suggestions for things that people did to make their on call Less stressful or more importantly the things they did for self-care I will say that I did remove the ones that quite a lot of the suggestions were I eat a lot of junk and I was like well that goes counter to the slide I gave before so You know selection bias in my data, I guess So I really like this one. It says you know rest when possible, but any activity that is restorative So what in this particular case, maybe it's knitting or reading And Moving your body right because especially when we're sitting there, you know And I really like the sort of thing with saying no to other obligations During your rotation might be the time you have to say no to some other things get someone to cover for you On some other obligations you might have So tammy said, you know during really bad rotations. She just curls up with throw rugs and tea on the couch Maybe taping taking naps having food delivery Um, and then if there are rotations when there's no pages be eternally grateful So that's another thing you could do when you have celebrate A restful and uninterrupted on-call rotation And like this Matt Simmons said he tries to get a massage during the week that he's on call Running on a treadmill every time you get to adequate sleep and then also painting And then um Jeremy said, you know binge watching the netflix shows that his wife is not interested in And eating so so many cans of pringles. So I guess I did slip one of these unhealthy Suggestions in there and then again if all else fails the ultimate suggestion to self-care during on-call Is forget to put your phone on the charger accidentally have the phone on quiet mode instead of ringers and Thank you, Jeremy for for that one. So As always, you know, Jeremy is is a thought leader in in all things so Remember, you know resilience strength is the opposite of helplessness right being resilient doesn't mean Being resilient means that we don't see things from the perspective of things that happen to us It's a matter of what we can do going forward And remember that a culture of blame creates a culture of helplessness These slides as well as some supporting links and resources Will be available at this url. I They might be there now, but it's a slightly older version of the talk So I'm gonna as soon as this talk is over. I'm gonna upload the updated slides And you can always find it there. So Thank you Could probably have a few minutes if anybody has uh questions Uh, I would also encourage if you have suggestions for Self-care around on-call that we didn't talk about that you'd like to share would be super great Isn't using the term postmortem Kind of flying in the face of trying to make it things less stressful, you know, nobody actually died usually so Let's forget that I work for patreon for a second So I personally I'm not a big fan of that term I I personally like after action report better or something we in our product Call it postmortem. I also similar to devops engineer and devops team and director of devops have decided there's certain hills That i'm just not going to die on But it can so the question is if it actually Does cause additional stress Then it can be counterproductive. Uh, that is a much much bigger and nuanced Discussion which could go into a whole other thing if it was up to me And I was creating a process by myself. I might not call it that But I also Would be more concerned with actually the process than the term But if you do find that that's causing additional stress in your organization That might be an area of opportunity to Investigate because words semantics Can influence how we think so to really enjoy the talk like a lot of the concepts I I feel like i'm trying to approach these problems from a less on-call centric perspective in more from Situations, I I guess I call it process scarring where a thing happened years ago And the organization organization changed what it does as a result And nobody has had the impetus or motive to go back and revisit why that is done Or there's that like there's that anecdote about like making a pot roast and always cutting off the bottom half of it And I think grandma why that was done. It was grandma didn't have a pan that was that large I'm curious like I really like the concept of organizational EMDR and game days was a great suggestion I'm curious if you have any thought or suggestions around stuff that's Not quite so on-call focus, but revisiting ongoing critical stuff So I think that's a that's a that's a good point when we think about how do we approach this continuous improvement opportunity when it's not directly connected to an incident or an outage And a place where that comes in is again doing things like Postmortems we don't necessarily postmortems on incidents and outages and they have to be fully processed So a lot of times we do and and there may be something to bring up and say we need to go and reprocess We have some trauma in our organization I have a similar example a company I worked at when I came in After a while I said why does it take an hour for us to replicate data from the back end to the front end? And they said well because we had you know half a dozen front end database servers and I was kind of looking and like why do we stagger right? It was like we would replicate to one front end database server And then 15 minutes later we'd replicate to the next one and the next one Well, you can probably guess where this came from is once upon a time Some developer wrote some bad sequel that dropped everything and it instantly went out to the whole front end and everything went terrible So what did we do instead of figuring out? How can we prevent that from happening? We said well, we're going to mitigate what happens not if but when these stupid developers break something Rather than saying how can we go into that? And so what when when those things get discovered there are areas of opportunity for an investigation to say like okay Can we go back to a better way of doing this because what we need to do? But this is where the storytelling comes in right it comes in we have to actually explore it right? It's not like well this seems dumb because you know what there was a good reason At any given time these decisions we make that we look back at in hindsight and say this is ridiculous It was the right thing to do with the information and the resources that we had at the time the thing is That was Technology we have better tools so we we can go back and revisit those things And those are sometimes really uh one of the things we find this again. Okay. I'm just going to keep talking about page duty I'm not talking about the product. That's good We you know we have once a month hack days and we there's some folks in our engineering that what they spend their time doing on hack day is actually looking for stuff like that Which is saying like hey, I did some sniffing around and I'm finding Things that just don't make a lot of sense So on this hack day I'm going to spend it as a research hack day And I'm going to try to understand why and then maybe on the next one I'm going to try to put something together to kind of remediate around that So if you can find things within your organization that allow for that exploratory time And have it be something that people who are Interested in that because some people are just not going to be interested in making things or that's okay They're not bad people. We still love them But some people are going to say like I really want to understand why things happen the way that they did And giving an opportunity and hack days can be a really great great time or your 20 percent or whatever You have to have some kind of exploratory time in your organization for that Thank you Um one thing I see a lot is when there are problems and with teams That finger pointing can often start and teams start blaming themselves Yes, this has a long-term corrosive impact and it goes past whatever the immediate Problem is and sometimes involves hr Do you have any practical suggestions on how to avoid scenarios like this? Sure So that's a really interesting point Which is that a lot of times when we think about about a blameless culture We mean we don't point fingers at other people, but just as equally we don't point fingers at ourselves And one of the ways you can do this that can really help Is celebrating failure? So uh at etsy They had this thing they called the three-armed sweater award Which is I think was given out once a month and it was basically to whoever fucked up production the worst But it was celebrated. It wasn't like a dunce cap. It was like hey because you know what The reason that happened is because you were trying to do something and it was it was a celebration of failure And I'm not saying necessarily do the exact same thing The three-armed sweater award so um I actually saw a tweet that rin daniels who used to work at etsy She uh has a talk she's given that talks about basically the time she completely hosed apache and Earned the three-armed sweater award and how that celebration of failure was helpful So there's and these are the things that have to come in at the management level and the senior executive level which is that We we celebrate failure because it means we were trying, you know charity majors has talked about this before and said if you Have been working in an organization for more than six months and you haven't broken production. You're not trying Right now that you're trying to break things, but you might be being a little too cautious So we need to look at and say like okay this thing happened. So first of all we want to understand How can we help prevent it from happening? But it wasn't the fault of the humans and in fact, it's actually we celebrate it We say like because we were trying so just sort of the way that we approach failure Can help people understand that it's okay and to not be Self-deprecating about it and not say like oh We were going again that also goes back in those cognitive distortions which are because we had an outage our team is bad Right, that's just it's it's part of that realization and and it's it's an unfortunate nature of The role when we think about system administration and sRE and things like that. It's one of the very um Uncelebrated functions in an organization similar to your corporate lawyer Nobody knows All the times that your lawyer kept you from being sued They only pay attention to the time that they didn't keep you from being sued Right and usually if people in the organization are aware Of system administrators or sRE or ops It's in the negative. It's in the time that they missed so looking for ways to celebrate When these things happen and say like okay, this is this thing that occurred and here's what we did And this goes back again into the storytelling of the post mortems and the after incident response Which is it's more than just this thing that joe did Right, what's the whole story around this so then it also helps kind of dilute it away from being like hyper focused On the one individual who happened to be the inflection point of what happened But there were many many contributing factors, right? It wasn't just what joe did that was just part of it Just to comment I always wondered how somebody could be a goal lead because I always just thought that was probably the hardest job on the team Oh, yeah, you were the one that missed it or you didn't catch it Oh, yeah Yeah So maybe have time for one like five minutes one one one or two more and if not i'm Findable on the twitters and i'll be around for another hour or so No, great. Well, thank you very much everybody. I really appreciate it and have a good last fuck of the day of the year Alrighty killer tomatoes So there's no no tomatoes today just avocados So there's a children's book in there somewhere There you go, there you could be is this type of avocado does this this type of avocado does this yeah, yeah All right Well, it is 432. So I'll get started so that we don't run late because I know I am the last session So thank you so much for coming because I know it's been a long four days Um, but my name is mary singval and I am here today to talk to you about developer relations Um, I know a handful of you throughout the room are familiar with the term is everyone here familiar with the term Yes, kind of maybe Okay, cool. Well today, we're going to approach it from a slightly different angle than you'll hear at most conferences I'm going to be explaining the business value of developer relations using the suit famous throughout california the avocado So why avocados you might ask um, and if you look at the screen There are definite resemblances and their intentional resemblances between me and the avocado But besides the fact that I find them really delicious and I can say that because I liked them before avocado toast took over san francisco But devrel has become known as the good kind of fat in tech circles This is an analogy to picked up steam three or four years ago when I was at a company called spark post And one of our project managers had a hard time saying developer advocate And so she'd get talking quickly and giving us instructions and it would come out developer avocado And my co-worker adrian loved avocados and my clicker technical difficulties apparently Right in mouse go There we go The co-worker adrian loved avocados and very willingly took on this mantle of developer avocado without much prompting And as he added to the team we decided not only is it kind of a cool team name But maybe we should make an analogy about it as well So that some of our co-workers have a better understanding of what we do So we decided to own this term internally as well as externally and an analogy was born and i'll be sharing more about that analogy today But for those of you who don't know me you may be kind of asking who I am between me admittedly kind of creepy avocado faces that have been up there Um and before I get to that I know better than to try and introduce myself before I introduce my sidekick ember Because I don't have people's attention until I do so this is ember dog His full name is ember dog pups McGee captain underfoot the third and you can find him on twitter You'll notice I changed it from my twitter handle to his twitter handle because as you do Um, but he's a medical alert service dog for me I'm a type one diabetic and he lets me know my blood sugar is starting to get too low But that's a whole other talk about monitoring and performance that we can give another day So Who am I? As for me, I have a journalism background But I entered the journalism industry right as most newspapers were laying off all of their writing staff So I have fantastic timing And like most tech companies I wound up pivoting So I'm now using feature writing and storytelling abilities to show the business value of building developer communities But I've personally worked with various developer communities for more than 10 years now at companies like O'Reilly media Chef software and spark posts Some of you here today know me from times that I spent running the booth at scale for any of those companies But my favorite thing about working with communities has always been figuring out What makes each of them tick as well as how to find the best solutions for any of the problems that they're facing And as I worked with all of these various technical communities I found myself going back to my journalism training Recognizing patterns during conclusions telling stories about how these communities were not only beneficial for the technical people in those communities But also for folks who were trying to understand new concepts or products As well as the companies that were taking the time to invest in these communities And it started to become clear to me that while some companies could absolutely succeed without a community behind them The best companies and the ones that were the most successful Were the ones who had communities who were loyal and were consistently praising the company without any sort of bribery Because they were the ones who took the time to invest in those local technical communities around them And I started to realize that most companies didn't actually understand what this true value was right They understood that developer relations was kind of a thing and community management was kind of a thing But didn't understand how to translate that back into the business value And for the most part these community teams were underestimated and are utilized and overworked We also lacked the resources to know how to make things better on our own as those community managers So I left the corporate world and started out on my own in 2017 with this mission And my main goal is to create resources about developer relations and community management Educating and providing professional development opportunities for those of us who are doing this on a day-to-day basis As well as their managers and stakeholders who are trying to figure out where they sit Within the teams and the companies that they're in And my hope is that by providing resources and working with these companies one on one The industry will slowly start to understand the business value of developer relations and the advantages that can next team Would these technical communities like everyone here at scale can give them in the same age So these logos on the screen are some of the things handful of the resources that I have provided and helped to provide As a part of this goal The one up there on the left is the book that I wrote that came out last october I have a few copies with me here today and a lot of what i'm talking about today comes from the concepts in that book Um, so if you want to talk to me more about that afterward, that's awesome I also have a discount code that'll be up on the last screen But anyway all this to say I do a lot of things I have a lot of side projects and i'm very involved in the communities around me But i'm incredibly passionate about driving the developer relations industry forward and working with companies to make sure that both the Communities that they're building and the people who are building those communities are set up for success so Now that you know the basics of developer avocados and you know that it's grown from there What does this all mean for you today? In short, it means that I understand the confusion around developer relations firsthand These days most companies have developer relations or a developer advocate on their checklist, right? We hired that person check that box cool. They're on their own. They know what they're doing I don't have to deal with them. It'll all be fine But then they get confronted with that. Why have like well, but why is this person here? And we're spending a lot of money for them to do a lot of things and the board is now questioning it And they don't know what those answers are and i'll be the first person to tell you devrel is not an easy one-size-fits-all Silver bullet to fix the business and get all developers everywhere to suddenly love your product and join the ranks of your customers It also doesn't fit within our traditional business value metrics that we're used to addressing But with patience and by asking the right questions devrel can be an incredibly valuable asset to Any business that has a technical audience and can sometimes even be the difference between success and failure So the principles that i'm going to be presenting today not only illustrate why developer relations are like avocados But it also shows you that by applying these principles devrel can be a valuable and healthy part of your business And before we jump into principles. I always like making sure that we're all on the same page with definitions So what do I use when I use the word community? It's this definition a group of people who not only share common principles But also develop and share practices that help individuals in the group thrive And how you define who falls into that realm of community at your particular company depends on what your intentions are We'll talk about more that in a minute But in general community includes your current customers as well as prospects As well as anyone who could at some point in their career be interested in using your product It's a pretty broad umbrella term So how does this fit into developer relations? First of all developer relations isn't just another name for a developer advocate role Developer relations is the umbrella term for the team whose primary responsibility is to build that community both online and offline That includes developer advocates It also includes maybe a developer events manager person Community manager can even go so far as to include roles like documentation and training at some bigger companies like twillio But at its foundation the purpose of devrel or developer relations is to build relationships with and enable these technical communities Devrel professionals acts as a liaison between the company and the community that they're working with Who are typically the end users of the product? And while most professionals have the best interests of the business in mind We as devrel professionals have the best interests of the community in mind because we understand that if the community succeeds The company succeeds But if the company sets the community up for failure There's a lot of times when you just can't come back from that and the company fails because the community knows that they're not being taken care of So this is the mantra that I use to explain developer relations on a regular basis To the community I represent the company to the company I represent the community and I must have both of their interests in mind at all times And if you saw my ignite talk last night, I talked about this there as well And this is incredibly important and it's a hard balance to keep But in order for the devrel team to actually be able to keep this balance They have to be fully supported by the company that they're working for so they need a clear set of business goals They need clear set of expectations. They need the right tooling But they also need to know that their work is seen as valuable And this is something that we struggle with because most of us who are in developer relations don't have a business background We don't have the communication skills to communicate upward to the stakeholders about how that fits into the business And where we're aligned with the company and all of those types of terms And so this brings us back to why I'm here today because I want to help you understand the value of developer relations and community building with the analogy of avocados So principle number one avocados are the good kind of fat, which you probably guessed based on the title of my talk Um, but devrel tends to be an expensive or fatty department, right? Because we have conferences that we're sponsoring. We have schwag that we're giving away There's open source projects that we're sponsoring or supporting often But used at the right ways in the right times with the right combination of items It can have amazingly healthy benefits for both your company as well as your community So what are some of these right times right ways right places combinations of things? It's things like talking to developers about product feedback while at conferences like scale or at events around the world Having one-on-one conversations with your top influencers and top community members who were really embedded in api field or an iot field It might be Being the bridge between those community members and your product and engineering teams and creating open source tools and Documentation around your product to help make the developers lives easier I call these connections made while doing this liaison work devrel qualified leads And what do I mean by that because if you're not a business person or a marketing person It probably doesn't make sense to you, but if you're familiar with marketing qualified leads anyone? With the term a couple. Okay, cool. Um, let me put it this way. How many of you got your badge scanned at a boost this week? Cool, you are now a marketing qualified lead for a company that you got your badge scanned by right So in other words, it's it's this idea of being a person who may at some time Be a customer for the company who scanned your badge because the marketing team got your information and they can rate your Chances of becoming a customer and then turn that information back into their sales team who will follow up with you at the appropriate time So marketing has now done its job of filling sales pipeline And I came up with this term after meeting with yet another team who had traditional metrics that were either given to sales Like how many people sign up for an account this month or to recruiting? How many applicants did we get in the last month or to marketing? How many leads did you get at that conference, right? And these are all things that devrel community professionals have zero control over Who knows whether the person that you met at your booth is going to wind up hitting it off with a hiring manager Or whether they'll actually get to the hiring manager because maybe HR has a policy that you didn't realize that they have to have specific College requirements or things Or if they just click with the rest of the hiring team, right? But whatever the case may be you can't be held responsible for whether or not that individual got hired You have no say as far as salary compensation or any number of other negotiating factors Or whether there'll be a good fit with the other team members So the idea is we aren't necessarily responsible for whether they got hired, but we can make those handoffs, right? We can say we introduced this person to the company. So what does this look like? Well, maybe you've encountered someone who's answering questions on your forum consistently And has obviously had a really good experience with your product They may be a great contact to pass off the marketing for a case study Or maybe they're interested in turning some of their longer posts into a blog post Maybe you're getting exceptional feedback from a technical individual and passing them directly to products might be a good idea They might be able to have longer form conversations to communicate some of that feedback Or partially important pieces rather than you playing the messenger between those groups Or if you're getting close to rolling out a new feature, maybe they'd be great for a beta tester Perhaps a community member has stumbled upon a really hard to solve bug And you trying to play messenger again between the community member and the engineering team isn't working You can pass them off to the engineering team And maybe that community member is even willing to help figure out what the root cause is of that bug and get to the bottom of it Or maybe you ran into a different developer advocate at another company who's willing to help build out an integration To help customers use your products in tandem These integrations can be huge because it can make it that much easier for your community to hook in and use your particular API or your particular product And on occasion will come across community members who just Get it right they click with everyone at the company. They understand the product They're passionate about the cause and the problems that you're solving They're already contributing during their free time. So when a position opens up, they're the perfect person to pass after recruiting And lastly of course sales Depending on the person you've met in the community It'll depend on who or that will determine who you pass them off to internally, right? Whether you start with a solutions architect or solutions engineer, whoever your technical sales person is at your team Or if they go straight to the enterprise sales team, maybe because they're the manager of a team and they're the actual decision maker But sometimes the handoffs won't be with the community member you've engaged with but maybe with their manager But you get the idea these connections are incredibly valuable and might not ever have happened Were it not for the dev rel team's direct involvement with the community who knows and trusts them So these dev rel qualified leads are incredibly important to keep track of for a number of reasons The most obvious one being of course that it's the definitive way to Attribute value to the activities that were responsible for on a day-in-day out basis But in aggregate aggregate It's also a valuable way to see which activities overall Are more effective than others in the long run as well as track themes throughout the industry So they have a tremendous amount of business value But they also contribute community value Because as you're making these introductions between community members and your co-workers You can also make introductions between community members who might be a good fit Maybe they're both trying to solve the same problem with their company Maybe they're both dealing with similar problem areas with your api But whatever it is, maybe they're passionate about the same project, right? But you're making these connections both internally as well as externally And this leads me to my favorite analogy for dev rel Thanks to my good friend Amy Hermes who some of you might be familiar with But this is that developer relations and community management is a pseudonym for cruise director And how many of you have been on a cruise? Okay, a handful and you know the person who kind of makes sure that you're feeling good about everything And you're excited about all the things that are happening and that you always have someone to talk to you And you're always involved in the activities Because you're not feeling left out that way That's what we do, right? So all of those people that I mentioned that are pursuing that new topic I as a technical cruise director am responsible to introduce them and foster that relationship to make sure that they're not only pursuing that topic And reporting back to me, but that they're also doing it together in community with each other So I can introduce Marie to Bob because they're both interested in the latest release of python And we're using that to solve problems within their company and our api is built on top of python And here's the benefit there. So I take a step back I let them have that conversation and then I can follow up with both of them separately and say hey How did that go? What came out of it? I'm so glad you were able to connect our things continuing to build that that particular segment of our community, right? So why does this matter? It matters because of the definition that we started with for developer relations that at its foundation The purpose is to build relationships with and enable our technical communities And this enablement is beneficial for both the company as well as the community I love this quote from zanmark and recent blog posts. He says enabled developers are productive They're less likely to churn and they're better suited to champion our products and services inside their teams organizations and wider networks And this is huge because once they start talking up your product outside of just talking back and forth with you Then suddenly they're there. They're your external developer advocates that you compensate with appreciation and community and Showing them that you're willing to amplify the things that they're working on as well Twilio's evangelism team puts it this way They say our job is to inspire and equip developers to build the next generation of amazing applications This means understanding what they're trying to do Pointing them to the right tools and training and generally helping them to be successful And this is not an inexpensive endeavor, but it is worthwhile And when twilio was first founded they were told they didn't stand a chance for the developer focus strategy But they went on to land a one million dollar seed round from a group of angel investors Including people like chris saka and mitchka four and one vc And the first full-time employee that they hired danielle morrell built at what is now Now acknowledges startup world's best most effective developer marketing program And she is now an investor and startup founder in her own right And they've got customers like del and twitter and lyft and sales force and huge companies right and they continue to developer They continue to cater to developers and build out what's now known as one of the best dev rel teams in the entire industry So they've invested a significant amount of money in something that they were told would never be a successful business strategy Because they understood the value of developer relations And if you can prove to developers unequivocally that you not only want but will listen to you and implement their feedback You'll gain their loyalty and they'll continue to give you that value day after day Principle number two avocados take on the flavor of things around them So there was a time when I didn't understand the appeal of avocados, which I know some of you in the room might agree with We'll see if we can change your mind on that But they're you know, they're a little bit smooshy. They're kind of bland They're somewhat odd looking fruits the masqueraded vegetables And then one day my aunt introduced me to this super simple really good lunch And she just took a white cheddar rice rice cake put avocado on top put a slice of already cheese on top of that And I later experimented added a fried egg and it just makes this really nice combination of flavors Because the avocados kind of take on the flavors around them and they're a wonderful agent of flavor as well And now that I'm hungry. I don't know about you, but what's my point, right? So in this same way developer relations teams tend to take on the flavor if you will of the product that they're working with We can be fluid in goals. We can be fluid in what department we're in And what the group looks like all depending on the needs and the goals of the company So what i'm saying is there's no one-size-fits-all solution What has worked amazingly well for twilio and gethub may or may not work well for you And this is a huge point of caution this meme was popularized by Matthew Ravel last year of hoopy And I think it illustrates this point really well. So it says we check what twilio did We copy it on a lower budget We buy beers for developers And then dug and marketing asked what our team is for the ceo notices and the team is disbanded It works so well, right? And it's funny, but it's also true and it's unfortunate when that happens because it happens far more often than I think we realize And when you look at someone else's strategy with how they're doing developer relations You're looking at a plan that they've developed over months and years Which is specific to their audience with their company's strengths and minds, which means it's not likely going to work for you This tweet from cmx hub founder david spink says it. Well, he says if you try to start things at scale They won't work and it was a quote from someone else and then he says it's a huge lesson for every community builder Don't replicate how a successful community looks today Replicate how they started small and focused I was waiting for you to grab the picture So the key here is figuring out what works for your community Both for the community you have right now as well as the one that you're liking you're wanting to grow into And there's two questions that can kind of help you figure this out from the beginning first off is Why do you want a community? And the second one is what do you hope to accomplish with this community? So first up, why do you want a community? And you'll notice I didn't say do you have a community is the first question Because whether you spent the time to build up a community or not Every established product has a community of customers both current and potential Simply a matter of whether or not you're choosing to be active and engaged in what they're talking about online So your why determines how you choose to interact with them going forward If you're familiar with simon cynic anyone? Couple nods cool. So this question won't be news to you at all then right? So for those of you who aren't as familiar simon cynic popularized a Ted talk an idea about always start with why right because if you don't have the answer to that why You won't be able to set yourself up for success when you're trying to accomplish your goals So our why drives how we respond to people and it drives what we actually say as a result of that This also drives the structure of who handles the team who handles those decisions And who's responsible for which aspects of the community initiatives And how we respond to community members where we send people for more resources everything else So why do you want a community the why doesn't have to be quantitative metrics? It can be abstract ideas But it should be aligned with the organization's goals and explain the purpose behind each community related endeavor at the company So you should be able to when you're creating a new initiative say we're doing this because This initial reason that we came up with years ago, right? It always points back to that north star of your why You'll also want to make sure that your why is driven by a reason not a result For example to make a profit is a result of growing relationships nurturing leads having a great product It's not a reason to be doing what you're doing And your reason for establishing community might be generating engagement making a better product or a better developer experience Or customer retention But the result of that Maybe those devral qualified leads that i test on earlier their opportunities to further product feedback pass along possible recruits Facilitated sales opportunity or more But your ultimate why needs to be your reason and your motivating factor for building that community in the first place So the second question what do you hope to accomplish with the community helps us figure out what devral looks like at your particular company This question helps you determine whether you're trying to create a community of customers or simply define a market And you might be looking for a core group of customers just to get periodic feedback from right and that's fine You can have a small super productive super influential community It also might be that you want to set yourself up as a thought leader in a relatively new space Maybe you want to follow trends within your niche in technology And you're relying on those connections in your community to help keep you up to date on those developments And the answers to this question will help you determine now how to best serve your community But also what department is the best fit for your devral team So if they're writing a lot of content if you're creating a lot of resources for community members Maybe they fit best under products If you're speaking a lot of events and educating people about the importance of where your product fits within the greater Technology industry, maybe marketing is a better fit But if they're working on sample apps developer experience, maybe engineering, right? So you can kind of take a look at both the why as well as the what and figure out what the best placement is there But placement of a devral team is a whole other talk and I can go into that at length So the key here is figuring out not only where your team will best will most succeed But also where they'll be be able to bring the most success to the community as a whole So let me play devil's advocate for a second because there's a somewhat valid question here If you aren't familiar with the value that devral brings to the table Why do we actually need a full-blown devral team to get feedback and improve developer experience because that could Arguably sometimes be done with a combination of product or marketing surveys engineering support technical writer All of the types of things and this leads us back to the mantra that I said at the beginning of the talk To the community ever send the company to the company ever send the community And this is what sets devral apart It makes us uniquely able to fulfill that relationship building and listening and understanding that goes hand in hand With building that community of loyal customers Because our primary focus and our primary goals are first and foremost based around the community And this focus and attention gives us an opportunity to build up that trust within the community, right? Because if they know that we're asking them for feedback so that we can advocate for that internally at our company They're far more likely to be upfront and honest with us if they think we're just asking them for feedback because you know We're checking that box They're not going to be as open with us because they don't have that relationship with us Authenticity breeds authenticity and while it's entirely possible for products and marketing professionals to have this viewpoint as well Their priorities are often split between feature releases lead generations other things on their plate Whereas again our focus is first and foremost always the community Principle number three avocados go well with many different cuisines, right? So from Mexican dishes to breakfast plates to classic blt You'll find avocados in all sorts of cuisines today And this couples nicely with the previous point that devral teams can take on the look and flavor of the company that they're at It's not going to look or feel or taste the same at each company But not only can devral goals and the department that they report to be different But the tactics that they use to create that community and engage that community Changes depending on the circumstances and type of product that they have Just like a good chef experiments with all different sorts of flavors All different sorts of ingredients to figure out what works best for the cuisine that they're making food for So developer relations teams do the same thing. We pull lovers. We try different experiments We check in with the community to see how something's working for them Because we want to observe the outcome and make sure that we're making the best decisions and the best choices for them It's just something that the community team at keen the new and understood well And this is their team mission from a few years ago and it says We use our superpowers to help keen.io grow into a sustainable business by supporting other teams within the organization Our internal community in accomplishing their missions and helping our customers partners investors advisors fans Friends and family etc our external community be everything that they dream to be And this makes it clear that they want to do whatever they possibly can to support Both their internal community the business organization as well as their external community Right Giving that communication from the community back to the company from the company backs of the community Because in serving both of those it brings that much more value to both of those groups And again more often than not this is most often achieved through connections If you're looking for a silver bullet the one simple principle you can take back with you today I suggest this focus on the what is it that only dev rel can do And there's a reason why you'll find dev rel teams over You know product and engineering and marketing all over different places of various organizations because there's a lot of overlap But again the one thing that dev rel really excels at is making connections for cruise directors, right? So we're the people who are able to take a step back observe the conversations that are going on Observe the patterns and then make those connections whether community member to co-worker Worker to community member community member to community member But then we make sure to follow up with those folks like I said earlier Are there other things we can help with is there other feedback that we didn't get from them at that point Anyone else we can introduce them to to help further their careers Anything anything we can do to solve any of the problems that they're facing on a day-to-day basis And in doing so we become the hub of the wheel, right? So we're essential to keeping that wheel of progress turning But we're also the connection point between all of those departments internally as well as again external community members back to our co-workers principle four Avocados take a long time to ripen So this is an even longer process if you take into account the fact that it takes five years for an avocado tree to even be fruitful After it's starting to grow, right? But once the fruit is ripe, it's not only delicious, but it yields a good profit for the farmers Avocados are kind of expensive if you hadn't seen that Um, but if you hadn't guessed it already or if you haven't heard me say it three or four times in this talk Devrel is not a quick fix, right? It's not something that you can implement and the next day your product is going to take off and be viral and be the most popular thing on the market It's a long game But it's something that with a good upfront investment and careful nurturing the final harvest can be incredibly rewarding for your entire team as well as the community the problem with Tracking only work output and this is something that I encourage people to to do Um, because this tends to be I feel like I missed a slide here, but it's not there. So I'm going to keep going um But the the one metric that people tend to track as a result is work output, right? What are you doing every day and that that works for engineering teams that works for product teams? Like how much have you gotten done? How much progress have you made on these projects? And the problem with tracking that for dev rel is because there's typically so much that goes into a single achievement A conference talk for example that it can look like your dev rel team isn't getting all that much to accomplish on a day-to-day basis Because a conference talk isn't just a conference talk Is any one of the speakers here at scale will tell you, right? It includes researching the right conferences finding the ones that are the right audience the right location The right impact and then you cross check that conference list against the conferences that are still checking cfps And then you write and you revise your cfp and then you hope and pray the cfp gets accepted And then maybe you're lucky and you get into the conference And then you actually have to write the talk which if you were here from maddie's talk You understand because he said that it's like oh shit now I have to actually go do the thing that I said I was going to do when I wrote that abstract, right? But then you have to write the talk and practice the talk and depending on your company get the talk approved by your corporate team And then you book travel and then you travel to the conference And then you give the talk and then and only then can you say hey? I spoke at a conference I checked that thing check out my workout, but it's done after six months of time, right? And if you're lucky and only if you're lucky No one asks you what that impact was on the bottom line for the business Because otherwise people start to see well But you spend a lot of time on that project and then you book travel Which costs a lot of money and then a lot of conferences are in fairly expensive cities So that costs a lot of money in and of itself And so they wind up seeing us as this person who just spends a lot of money flying in the world to exotic locations like pasadena, california And we're parting with the community and not actually accomplishing anything According to the people who are seeing our work output So we need business alignment to have the ramp up time to be able to succeed on these longer commitment timelines Which building a community frequently requires So what is the better metric? Tracking connections or you can probably say it with me by now devrel qualified leads, right? Because here's the deal when your devrel team is out at the event They've spent so much time preparing for they're going to be meeting customers They're going to be meeting potential customers. They're going to be meeting influencers throughout the industry It's what they do. It's what we're good at making connections and being those technical cruise directors Like I mentioned earlier And if you encourage them to then pass these people back into the company making those appropriate connections to sales product marketing engineering And if they know that they can trust the rest of the company to handle those connections with the appropriate amount of respect and attention Then the company is enabling them to do what they're best at and what only they can do And turning that into a metric that directly proves the value to the company Well, continuing to allow them to serve the community in the best way possible And when these opportunities do ripen down the road as they will When the developer is currently at a tiny little startup He didn't have a need for your product is now working for a huge enterprise company That developer knows and trusts your devrel team. They know and trust your product team or their engineering team And suddenly that new company that new huge enterprise becomes your biggest customer But if you put a metric of sales or recruiting or any ROI goal that needs to prove an immediate return on this team You're setting them up for failure because they won't be able to do what you've hired them to do which Goes back to the mantra right because you've hired them to represent the company to the community And take the feedback from the community internal to the company And if they're responsible for the success of other department's metrics They'll be too focused on their goals to be able to effectively create those relationships on a day-to-day basis And they'll be too worried about losing their job because they can't make their numbers Right which should never have been given to them in the first place But by focusing on these devrel qualified leads, you set both the devrel team and your community up for success So to recap we've covered four different principles today First avocados is a good kind of fat used in the right ways at the right times the right combination of items They can be healthy for your community as well as your company two avocados take on the flavor of things around them devrel teams tend to work With fluid goals and fluid departments and things like that depending on the goals and the needs of the company that they're inside Three they go well with many different cuisines And like I said like a good chef experiments with different combinations of flavors and ingredients Devrel teams figure out what works best for their particular community by pulling a variety of levers and observing those outcomes and four Avocados take a long time to ripen. It's not a quick fix But with a good upfront investment and careful nurturing the final harvest can be incredibly rewarding And hopefully these four principles well, perhaps a little quirky Will help you remember that the true value that devrel can bring to any developer focused company in business I believe strongly as I said at the beginning that use at the right times in the right ways with the right combination of items Devrel can bring about the type of value that can absolutely transform your product and really make you the best product for your community that you can possibly be Could transform your entire method of doing business Why might you ask because Avocados are good for your heart Bonus principle and the more that we learn by researching and collecting data about avocados the more we're realizing just how good for us they are right And likewise with more research and data around devrel you realize that it's not only good for your business But in many cases essential to maintaining a healthy product And no one can deny that a happy community and a healthy product are good for the heart of every company Just like avocados Thanks so much for having me here today I've got my details up on the screen that discount code that my editor provided me with is up there as well I love talking about these topics. I love working with companies to figure out issues that they're having Particularly how to set up your devrel team for success So if you have in-depth questions about your particular company hit me up my dm zero open You can shoot me an email. I know we've got time for questions here, too and I have a few copies of the book so I could do that as a like whoever has questions first four people get a copy of the book, too But we've got some time so Yes, I was curious. Do you present metrics regularly and And how does that go over what kinds of things do you present and what format and then who do you target? Sure, so it depends on the company. It depends on the team I start with those devrel qualified leads mostly because that's the easiest one to kind of ease folks into Like you you're able to keep doing the things that you're doing you're able to keep implementing the community feedback But also be showing value upward to the stakeholders in the company As far as how often I encourage teams to automate as much of the metrics they're being asked for as possible Because if you find yourself two in the leads with the numbers you won't be making those connections that you need to be making so Do you have a general size for community when it makes sense to hire a single person or then start expanding to a team? So that's an interesting question. Um, I've had a couple clients lately that to be honest when they first came to me I would have thought they were too small of a company to start with that I have one client that i'm working with right now that has four full-time employees and one contractor Um, but they are intent on building community into the foundation of their product, right? And they know like sure we could do this for a couple years before we hire someone full time But if we're doing that then we aren't focused on the community And if we choose to focus on the community then we won't be focused on building out the product so rather than splitting their interests and probably Falling down on the community side of things because that's not what they're trained in it's not what they're used to They're wanting to implement things along the way So i'm working with them to figure out like how can you set this up for success within the actual product that you're building To set it up for success for the community But then also what are some of the infrastructure and things you can build along the way? so if if the founders of the company are community focused and are technical and really want to build in that technical community feel to the product I think there's some companies that absolutely can do it from the start and have um Twilio that i mentioned earlier danielle morrell was their first full-time hire so she was employee like Three i think so they built it from the very beginning as well And i think the most successful companies are ones who can do that and on the other side of that have you ever seen Giant companies have dev rel that's only internally focused not an external community for other areas is that successful or there's different considerations then I think there's some different considerations But I think it can be successful and the most successful teams that i've seen do internal community Are actually encouraging their engineers to be more outward focused so it's still It's still the dev rel person building the internal community and making sure their developers are are Being effective and happy in their roles, but it's also encouraging them to explore You know what you'd write a really good blog post because you're awesome at writing documentation Let me help you with that You might want to go speak at this meet-up because you gave a really great lunch and learn about that topic the other week Right and so encouraging their internal developers to be more externally facing And almost using that engineering team to be kind of part-time developer advocates on the side Yeah, brian, sorry apparently Hey Great talk by the way. Thank you So let's assume that i'm an engineer who works at a medium to large company that does not do dev rel And I think hey, it's a great idea What do I do? I'm not a decision maker. I'm not a manager. I'm not a vp of any sort. I have no budget I'm just a you know a button pusher of a routine sort What would what would you recommend that I take back to my company say to I don't know say my manager my director To try and say hey, I think dev rel might be worth pursuing. Yeah, you have a suggested like conversations are sure um So if you're actually interested in doing that I that's like chapter two in my book goes really really in depth But cldr is start figuring out what some of the best ways to implement that might be so Are you finding community members are really coming back saying we need more content or the developer experience is not so great And we need some help with documentation or with the the ux general ux of your api, right? um, so figuring out having some of those conversations internally can be Really illuminating as to what the needs are that we need to meet and that might be You know, hey, can I volunteer to be on Support for a couple days or can I go talk to them about what they're dealing with? Can I write a blog post for marketing? Can I go give a talk at a meet-up right and asking if you can take some of that Time of your own to do those things and I encourage you to do that as much as you can on company time And not investing it outside because I've seen so many engineers who are like no I work 50 hours a week and then I go speak five times a year and I write all these blog posts And I'm really burned out like no don't don't do that But seeing if you can figure out, you know some companies are doing more of the like 10 or 20 of your time can be community facing efforts or Tech debt or whatever those things are so seeing if there's some open source projects You can invest in because if you can start taking some of those things for a little test runs And if you write a fantastic technical blog post that does really well then marketing is going to start going Hey, we need another one of those can you do another one? And then it makes that argument a lot easier to go We could totally do another one, but it might make sense to hire a developer advocate Or it might make sense to hire a community manager or something like that It almost sounds like the suggestion is to do a little bit of dev rel on our own to kind of kickstart things Yeah, definitely definitely and that's actually what I tell people too who are interested in the possibility of dev rel But haven't done that because the one biggest difference that I've heard from Any engineer who has switched from engineering into a developer advocate role is that there's so much context switching And so many times when they're focused on a project And then have to get pulled into something else because you're not just coding right you're doing a little bit of coding You're doing a little bit of writing you're doing a little bit of speaking You're doing a little bit of a lot of things and so kind of getting your feet wet in that to make sure that stuff You actually enjoy is helpful Thank you. You're welcome So one of the metrics that he talked about was that's the easiest to measure is qualified leads, right? But but that might not work for example, that doesn't work great for us for two reasons one Qualified leads work well if the value of the lead is high, right? But if the value or lead Is like a SaaS product that you're billing at $10 a month, right? That's harder to kind of measure, right? It's also like getting three qualifiers at a conference is Is totally not worth it, right? Right or the other for example in our case It's an open source product. So it's an open source project that then leads into a product, right? So Measuring qualified leads on that is also harder. So what do you recommend for metrics? For for these kinds of situations Sure So the the interesting thing about qualified leads from the dev rel standpoint and not from the marketing standpoint Is it's not often that you're going to be able to put a price point on that? So I'm you know passing someone off to marketing to write a blog post I'm not going to say I think their blog post is going to get this minute This much traffic and this much click-throughs and this many people signing up But your qualified lead could be I found someone a new contributor And I'm now mentoring them which is encouraging them to contribute more, right? So it's more of a figuring out the way to frame What you're spending your time doing in a business metric that the rest of the business is going to understand Rather than here's how much money this is worth and so you'll start to figure out, you know I had 12 conversations on Thursday at scale These five were really really good and these other seven were fine But they weren't quite up to par and I don't want to necessarily say that Like here's the seven that I'm going to stick everything on right you want to pass those five along And then continue building relationships with those other seven But pass along the highest of the possible leads, especially when you're getting a program like this going To make sure that you can kind of build that trust and authenticity internally in your company as you're passing those people on Make sense. Yeah Rob you had your hand up I think So most of the stuff that I've read about dev rel seems to be very product focused Curious of you have either examples or information on services companies and Like their interaction with communities and how to do dev rel at a services company Yeah, so it's interesting Dev rel is definitely more popular in api tech companies or open source companies because it's a super easy like Well, you've got developer relations folks out there doing this and it attracts more developers and we're good to go, right? Um for services companies. I've seen people hand it a couple different ways. Um actually know one of the dev dev advocates over at american express And he's out there giving talks all the time and they don't they don't have american express doesn't have a developer platform Like they don't have any api. They're not looking for people to hook on and use their product and build on top of their product But it's almost more of a recruiting mechanism. It's a we're doing some really cool stuff behind the scenes over here And it's interesting to us, which of course sparks the interest and spreads the brand awareness and things about the The company as a whole, but it also makes people think when they're looking for a new job Hey, that's right. That guy donovan was talking about cool stuff that they were doing over american express I wonder if they're hiring and so it's it's less of a feedback focused role and more of a hiring or making connections in the community that could lead to Maybe customers down the road, but also recruiting or Um Content or things like that as they're finding partners to work with so that makes sense What advice would you give someone who's interested in getting Who's looking to get into like a dev rel role? Sure. Um, so it's an awesome time to be looking to get into dev rel because there are so many jobs right now It's kind of ridiculous I one of the resources I put up there earlier is dev rel weekly and it's a newsletter I put out every week and I include a link to a jobs board and I was laughing the other day because when I started the newsletter I was listing like well, maybe I'll list the open jobs in the newsletter itself And very quickly realized that there were like 90 links and we're up to now 150 links and I have Gallo who helps me out with a newsletter who goes through that once a week and cleans out any of their clothes And it's still consistently like 140 to 150 links per week that are going out So there's a huge amount of people that are hiring if you're interested Um, but kind of like what I was saying to brian like putting yourself out there a little bit You know going to marketing and saying hey I want to write a blog post I haven't written a blog post in years I actually had someone when I was working at spark post who I he was Gave a great description of something he was working on. I'm like cool. Can you write that down? He's like, I haven't had a blog since I wrote on log live journal When I was in a high school like no and I was like that's fine. I'll work you through it Like and we had a template to kind of show people Here's the points that you bring up and here's how you bring it all together and I'll edit it at the end And we'll work on it together. It'll be a group project but Kind of getting your feet wet in those places to see like do I actually enjoy writing blog posts Do I actually enjoy giving talks which I happen to know you do do I? Or at least you do give talks Do I enjoy going out and hanging out with folks after the meet up and getting to know people one on one And kind of getting your feet wet in those areas and the nice thing about that is it might be that You know what? I don't want to do this full time But I do enjoy connecting with the community which will make you incredibly valuable to your marketing team Incredibly valuable to your product team when you're giving them feedback your engineering team, right? Because you're doing those types of things Even if it's not your full-time job And so by getting your feet wet it also gives you an opportunity to build the relationships internally at your company And if anyone is actually looking for a dev rel job, let me know because I'm always happy to help Kind of put you in the right place there, too I'm coming from the opposite side. I have a community And now that I know about dev rel people I want to connect with them Because I want people to come and speak at my conference. Cool So how what's the best way for me to find out What companies have dev rel and who they might be? I mean, I can think of I've got two companies that I'm working with now That I know those guys, but you know, awesome. I want more people than just two companies So there's a lot of us We've been it's been growing in number over the past few years, especially I've been Pleasantly surprised to see how many more dev rel folks are coming out of the woodwork I pulled this slide up again because dev rel collective that one in the middle there is a slack team that dev rel professionals are on And so we list a lot of resources on the website for that with like here's various places. You can find information Or if you want to ping us and say hey, we've got a cfp or various things that are open We're more than happy to post that in the group Um Dev rel weekly again is a good place to kind of see who's talking about developer relations because I'm pulling content That's been published throughout the week. And so it's a nice way to see like Oh that name's shown up three times in the last month. Let me go follow them on twitter um, but there's a variety of twitter lists and things like that out there that are just a great compilation of Folks to follow in the industry and I'm more than happy to help you out with those if you shoot me. I know Cool. Thanks Anyone else? No going once going twice Cool. I have a handful of stickers up here. I have four copies of the book Like I said, I lost track of who the first four people were to ask questions Which I understand is my responsibility But if you were one of those first four or you want a copy of the book feel free to cup up and grab it Um, and yeah hit me up. Let me know if you have any other questions Thanks a lot