 How's it going? I'm just going to jump right into it. So my name is Omar Bensadan. I'm a senior product manager at VMware Tanzu. My Twitter handle is there, in case you like some of the stuff I'm talking about and are interested in finding out more about this sort of thing, I tweet about this sort of thing all the time. My job is entirely focused on open source software. I don't know a lot of other product managers who work in this space, like, primarily. And so that's kind of what I do on a daily basis. I worry about open source from a product manager's perspective. And I work on a product called Knative, which some of you in the infrastructure space may have heard of. And throughout this talk, I'm basically going to talk to you about kind of a case study of what Knative did over the course of six months, six to nine months, to basically completely revamp our getting started experience, and we improved satisfaction with that experience by two-fold over the course of three months. And we'll go over some of those metrics as well. But just to start off, I work primarily on Knative as a product lead in a working group called the Documentation and User Experience Working Group. Knative is creating a standard serverless layer on top of Kubernetes, and it's quite a technically complex product. It's literally like a buzzword stack on top of a buzzword, and that means that it's really non-trivial technically, right, like we're sort of at the edge of an edge. And when I joined a year ago, the getting started experience was really complex. It was not very opinionated. It didn't give you a lot of like guidance. It just sort of threw you a bunch of YAML files and hope that you figured it out. And I would argue that if Knative, which is a buzzword stack on a buzzword, can have a pleasant getting started experience, right, given the monstrosity that local Kubernetes installation is, almost any project can. We're often told with technically complex products, usually by senior engineers and especially by on-infra products, that they're just incompatible with simple getting started guides. Users or operators, right, like these are people who are very technically savvy and they want things sort of thrown at them, right, just a bunch of code thrown at them. And I'm going to argue today, basically, that we need to separate these two types of people. We need to talk about prototyping installation. We have talked about production installation. Those two things are not the same thing, and they don't serve the same purpose. And so what we're going to walk through today is this agenda. You'll see it on the screen a couple of times. I'm going to walk through basically five parts of this presentation. Not going to run you through it. You can read, but you're going to see this a couple of times and we'll kind of dive deep on each one and we'll start with why you should care. So I want to talk about the goal of your open source project and how getting started guide might relate to that, why you should listen to what I have to say about it specifically, right, like what makes me a person to tell you about this, and then I have some good news and bad news for you. So there's a lot of open source software out there, right, as you may or may not be familiar with, being that you're at an open source summit. And the most limited valuable resource there is, is developer attention, right. Most of times people are coming to your open source software project. They're not just like seeing it in a vacuum. They're comparing it to other open source software projects. They're comparing it to managed products, right. And so that means that getting people involved and getting people to understand what your project does really quickly means you get more developer mind share. If developers come in and they can in 15, 20 minutes understand what your project does in a way that's hands on, maybe they don't leverage, maybe they don't pick it up today, but the next time they're in a meeting and there's a, there's a business problem that your project can solve, all of a sudden they remember, oh, I did that getting started guide on that one website. We should maybe check this out. So when is it important to have a good up and running experience? I'm not someone who's like a getting started maximalist. I don't think every project needs to have this sort of thing. But there are a couple of kinds of projects that definitely benefit from it when you want to grow a base of users or contributors. I mean, who doesn't? It's really important to have this kind of getting started experience, right, especially with open core business models, like the idea of like you have like sort of open source software and then like you build stuff on top of that open source software, which the guy here in the red fedora definitely knows about. And it's extremely common in infrastructure. It's very common. And if you want to grow a contributor base, getting people to understand your sort of nurturing your user to contribute or funnel is really important. So getting users, the more users you get in the door, the more contributors you maybe get at the end of that funnel. If you want to grow a base of vendors, right, this is something that's like really common nowadays. It's like companies make these big bets on open source software and they say, okay, we want to create this platform for platforms and we want vendors to come in and plug in around the edges. And we want other sort of vendors to come in and create software on top of these platforms and getting people to understand the sort of to borrow from Jeff Bezos. The undifferentiated heavy lifting that your product does is super important right off the bat. Like getting those people to get like a bit of a twinkle in their eyes is important. So when is it not in the very early stages of the development of your product? Don't bother. It's fine. You don't need to do it. In fact, it's probably a distraction because the product is going to change from under you. Also, if you're a developer is in the stadium model of open source, which basically means you're like one dude on a keyboard, dude or dude on a keyboard and there's like 100 people around you cheering you on and submitting a bunch of issues and you're overworked, definitely do not create a getting started guide because that stadium will just grow, right? So only if you want more engagement, should you try to do these things? Okay. So why should you listen to what I own? We're been sit on specifically have to say about this. So here's some, some background. I'm the product lead for K native, which is has like a slack community with like 4,000. We have like a production serverless platform on top of Kubernetes soon, there's going to be a case study on our site that shows that like, you know, it's processing 6,000 transactions per second, right? In some instances, and it is the leading installable serverless platform on Kubernetes full stop. I authored or getting started guide myself after interviewing 20 plus users about their experience getting started. And I'm not a designer or a technical writer. Like I, this is not something that I like have been trained to do, right? I haven't been doing this for years, like writing documentation. So if I can do it, you can do it. And the results kind of speak for themselves. When we first started interviewing users about the kind of serving, getting started experience, they rated their experience of like a 3.3 out of five. We brought that up to a 4.8 out of five, right? And these, by the way, because I just watched a presentation on data and how to contextualize it. It's like seven or eight in each of these groupings, seven or eight people. And for the event and getting started, it went from a two out of five to a 4.5 out of five. Right? It's still not perfect, but it's a lot better than it was engagement on the K native site is up 10% since we made this, this getting started guide, right? And users who go through the getting started guide are two times more engaged than the average user on the site, right? And this shows up in the analytics. It was so awesome to, to see that happen. That's all great, right? Like that sounds awesome. Like we all want that for our projects. And the good news is if you do some of the stuff that I talk about, to some degree, it'll work. Like it's, it's a tried and true method of making good software, talking to people about the software, right? Like this isn't a surprise to anyone. The bad news here is that it's work. It does take quite a lot of resources. And the truth is it required a PM working almost full time on this over the course of like three months, right? You might not get to, to sort of creating these like massive getting started guides. And you might not have the, your product might not be technical enough to even warrant this much time on it. But there are very tangible things that you can do for your project that don't require as much commitment, right? And can yield you better outcomes. And so how I'm going to frame this basically is I'm going to talk to you about what you can do and then I'm going to show you an example of what Knative did as an example, right? So it's going to be a case study and then framed around that case study is like the action that you should take in order to, to, you know, achieve those outcomes. Cool. So let's start with up and running. The question is basically do you provide an easy opinionated local installation of your software? So this is a great way to test this. Get a fresh machine and see how quickly you can access the software that you produce, like go to your site and try to like actually use your software. How can you get it up and running as quickly as possible, right? You don't need to give users all the levers and knobs right away, right? We need to be separating production and prototyping use cases because how people assume people adopt software is they come to your site ready to adopt it. That's kind of the vibe I get from a lot of infrastructure software. It's like, here's how to deploy it in, in, in Amazon, in, in Azure, in Google, like here's how you do it. And if you want to do it on bare metal, you know, it's like all this, it's like, actually, I just want to know what you do and why I should adopt you to begin with, right? That's, that's kind of the point. So we have to separate those two use cases because definitely important to teach you about how to install you in production, but you might want to show off what it is you do well before that happens. The gold standard for this is a one line install. You show up, you bang something into the terminal, all of a sudden you've got the software, right? And even if it's a curl to bash, people are suspicious about curl to bash because, you know, you should be. But most of the time in user tests, probably like 20 percent of developers actually checked what was there. I think that because we were in a user test, they probably trusted that, like, we wouldn't be like, ah, gotcha and then hang up. So yeah, that works too. And it's also OK to be opinionated about your prototyping installation. Like, it's fine to make opinions. OK, you've got a hundred different networking layers to support. Fine. Just pick one for people, right? Like, they don't need to be making those trade-offs. And we'll talk about what that looks like for Knative as well. So take action, right? The thing you can do here is try to understand what your current install flow looks like. What are you assuming about your users that are coming to your site? Are you assuming that they're installing it for a production installation? Or are you assuming that they are using it for prototyping? How easy is it to install for prototyping? Basically, it boils down to, are you making your users jump through hoops? Or are you jumping through hoops for your users? Right? Are you making your users come and figure out how to put together a prototyping installation? Or are you figuring all that out, bundling it in a batch script and putting it on the site? Knative is built on Kubernetes, which is the granddaddy of complicated installations, right? Like, it works everywhere, but it's like very complex. And locally, it's very complex. So we had this challenge where it's like, how do we provide a good prototyping experience to a thing that runs on lots of different platforms, right? Like, and is is different in all these platforms. This is what it used to look like. Basically, we used to say, OK, have access to any Kubernetes cluster. This is, by the way, not Knative inventing. This is just Knative in general. Install Knative serving CRDs. Pick and install a networking layer from the five we offer. We don't give you any opinions on which pick and install DNS from the five we offer. We don't give you any opinions on which still inventing CRDs. Anyway, you have to make all these opinions, right? Like you have to have an opinion on what networking layer you want or what DNS you want or all this random stuff. And so this is our MVP. We were like, OK, let's like package this into a script. Like we know we like this, this and this works for prototyping. Let's make that a curl to bash. And that we got a lot of great responses on. But the one response we got was, I don't like curling things to bash. It makes me wary. And so we made a CLI plugin, basically, right? We have the KNCLI, you just install that. And now it looks like this. Right? We assume you have Docker and Kind already installed because if you're our target demographic, you usually do. And all you do is basically you install this package. You do KN Quickstart kind. You get this. Right? And like this is something that like if you see this in a terminal and a developer, like you're like, whoa, that's pretty cool. Like there's emojis in there. I mean, come on. So, yeah, and that and that like basically made it so that our getting started guide before was like, I don't know, a completion rate of like 3%. I'm like kind of imagining that because there wasn't really a good flow for it. Now if you hit our homepage, there's a 10% chance you download this and finish the guide, finish the guide, which is kind of insane, right? Like for web 10% is a lot. OK, so you got your up and running. You've got a way for your users to prototype with your software. Fantastic. How are you going to bring your killer features to the forefront? And in Knative we call this superpowers. Basically, what are your products features that are most likely to delight users? And what are your projects products killer features? As a product manager, when I got into the community, I was like, OK, what do we do better than anyone else? Right? Like what are the killer features of this project? Like why is this the leading installable serverless platform? And even if you're not the leading installable serverless platform, you just ask yourself like, what are the features that make us really happy? Right? And you're and because we're talking about developers here, you're probably going to there's probably going to be a diagram. And then how can you get users to an aha moment as quickly as possible? So an aha moment is basically a moment where the user understands your product and also is starting to think about other things your product could do for them. Like so they and this will become clear in the next example. So I'm going to save it also because time. So the thing you can take action on is just make a list of your killer features in no particular order. Things that you think would delight users like things that you're proud of having built. Right? That's the sort of list you want to make. So for kind of serving, these killer features are dead simple services, right? You can take a container, put it on Kubernetes, you have to deal with like the four or five different, you know, artifacts that you have to understand to do it, right? You just put a container in the terminal into the CLI and boom, you've got a service that scales to zero and you can do traffic splitting really easily. So you can launch one version of the service and another version of the service and like you can in a CLI command split traffic between those two things, right? So let's take a look at scale to zero's aha moment, right? Let's like like the aha moment we discovered was watching the pods actually scale up and down, right? So what we do is we give you this this like this container, it does a hello world, you spin it up, you curl the thing, it says hello to you. It's a very standard thing. But then we tell you, OK, pull up in a separate terminal, watch the pods, just watch them, right? And we say like if you wait a little while, you'll watch these pods spin down automatically. And that feels like magic, right? Because it's like I didn't touch the thing. Like I didn't do anything to have that happen. And it automatically like deprovision the resources that I have. And so this is the kind of thing you want to be you want to be asking yourself, right? Like once you have your killer feature, how do you showcase that feature? Like how can you show people how cool your software is? And usually the answer to that is the stuff that you think is cool. But like most of the time it is. OK, using progressive disclosure, this is really important, right? Deciding which features go where, you know, you've got your list of killer features. You maybe even have like thought about some aha moments, right? Now, how do I organize this into a comprehensive guide? Like I know these things are cool, but but what do I do to get people to understand how cool they are? So it's a bit more of an art than a science, but it's a bit of a science, too. Basically, progressive disclosure is the concept that you should be walking through concepts or functionalities that are increasing in complexity, right? And this is very, very similar to the sales technique of like foot in the door. If you agree to a small request, it increases the likelihood of you agreeing to a second larger requests, right? That salesman uses all the time. Door-to-door salesman comes to your door, knock, knock, it's hot out. You have some lemonade, would you like to buy a vacuum, right? It's it's it's very, you know, it happens really quickly like that. And then think of your getting started guide in the context of a user journey and be opinionated about that journey, right? Where do you want users to go? What do you want them to discover? And in what order do you want them to discover it? Don't throw your your entire organization at them as as a as a, you know, basically documentation site, like have an opinion on what you want people to be learning. And you can take action on this. So once you get your killer features, lay them out in order you feel would be good, right? Like, this I think makes sense to me. And I think I'm progressively disclosing like in a mountain complexity. For Knative, it looks like this. We said, OK, we know that deploying a service is like the simplest thing we can do for people. You just deploy a service, right? OK, watch it scale up and down. Now now people understand auto scaling, right? Make a change to your service. So now we understand that we can make a change to our service and that that changes stored in the system as a revision, right? And the next thing we teach you is split traffic between different revisions. I've basically been able to show you that this thing auto scales stores your changes and you can split traffic between them over the course of a couple of steps. But if I had done it in a different order, that wouldn't that wouldn't have been as clear, right? OK, so you guys are seeing this and you're like, whoa, this is like really great. Good, Omar, you're doing a great job. Like this looks seems like you'd be a really smooth experience for people. But I still don't really understand how you get to that. Like it feels like there's like, you know, I don't have a product manager in three months. Luckily, there there is a life hack for this. It's just talk to people. That's it. And I know it sounds really daunting, but it is the most fun you will ever have with your software. And I'm not kidding, like watching people use the software that you build is both thrilling and terrifying in like such a guttural way. It's like awesome. So I'm going to talk to you a little bit about like how you're how you're going to do that. I want to like full stop user feedback is king. Full stop like getting getting users feedback on your software is one of the only things that makes your software more valuable. It's just true. Sometimes there's like a guy in the back of a lab who builds like something incredible and changes the market or whatever. But I would assume you're not that guy just just because like there are very few of them. And and I can't even think of one even standing here now. But you can get by without it. If you do some of the stuff that I talked about up until this point, you're probably going to be better off. And if you didn't, but I really insist that you try to talk to users. So making your first prototype, you don't have to build the website. You know, once you have all this stuff together, you don't have to go out and build your documentation site around this right. Like spend hours trying to pick a framework or whatever. Right. Just make it anything. Make it a Word doc. Make it a GitHub read me. Make it a PowerPoint. It really doesn't matter. Your prototype could test either your installation that you just tried to make if you had a complicated install before or it could test your showcasing of your killer features, a.k.a. You're getting started guy. Your prototype should be complete enough for someone to follow without your instruction. And this is really important because when you give users something to follow, right, like like you are not going to be on a video chat with the people who visit your site. So you want to be able to see how people are actually going to use the stuff that you build. And then your prototype is just a prototype. Like don't try to nail it on the first try. Feel free to leave errors and typos in there. Like it's going to be fine. Right. Like just let people know it's a prototype and you're testing stuff out. Like try to make this as easy on yourself as possible. There is one thing I will insist you do, though, if you're going to talk to users, you need a user interview script, right. You cannot have a casual conversation with six different people that go that goes a hundred different ways and come out of that with anything but confusion, right. You've got to be able to say X amount of users agree with this statement. And the only way you're going to get there is if you can create a user interview script. And I'll actually provide a template for that at the end. So split your script into distinct sections with distinct goals, right. Great ones to start with are the background, get understanding of who the user is, value proposition, get understanding of why they find your software or your category of software useful, right. Just broadly and then actually test the software, right. And some of the stuff in that value section like will surprise you. You're like, I thought people use my software for X. Turns out they use it for Y, right. And I think that's a really interesting. It's a really interesting to hear what people say about that. You always want to include a background section because it might be able to, you can, you can ask questions that may help you make assumptions about your end user. For example, for our Quick Start, we chose kind. You have to have kind installed to do this Quick Start. And that works for the 80% of users that come to our site that are like Mac users, right. And most people we talk to. But whatever we happened across the Linux user, it wouldn't work for them. And we found out that actually MiniCube works better. But we only knew that because we were asking background questions and we saw like, oh, we've failed with every single Linux user. Maybe kind isn't the best option for them, right. Because it turns out Docker on Linux is kind of screwy. So yeah, it's really important to ask those kinds of background questions. This is really cool too if you, if you have a community to share stuff with as often as possible, try and make your qualitative questions quantitative. Like the really important ones like remember earlier in the slide where I was like we took it from like a 2.4 to a 4.1 or whatever the numbers were like the only reason I was able to do that is because those were the metrics I was chasing. And I'm able to say hey look like we actually made this tangibly better. Using a number scale like allows you to basically make before and after comparisons like did I do better than what was there before. And then also allows you to compare between users. So there are some users who are extremely expressive and it's so funny to do this sometimes because you ask the same question like three people and they respond completely differently. But then they're out of vibrating is the same. It's like it's really, it really is funny that way like people have a huge variance in like what they think is actually a big deal and what they're going to tell you they think is a big deal. So this is a basic outline. I'm going to give you at the end like like a link to a Google Drive with all this stuff in it so you can like use it. You can sort of treat it as like you're you're getting started with getting started guide. So you can see here like in the background just tell us about what you about yourself and what you do. Have you used our our software for how long value prop what were your motivations for checking out our project. What problems were you trying to solve with the project. Whether tools or software were you looking at to solve those problems right so you can like learn from other competitors in the space. And then did that did our project solve the problems you were hoping it would solve. Right because that could show you like are we marketing this incorrectly like like is the home page saying something that is making people expect something different. And then the user testing it'll depend what you what you make right but the one I would focus on is how would you rate the current getting started experience for one to five and then how would you rate like what you just saw. Right like like you basically ask them to rate what they what they the getting started experience they had experienced right so like how how like how would you rate one to five getting started with our project. You walk through the user test and then you ask them the same question about the user test they just did. Yeah and again all this will be in a Google Drive folder at the end. Okay this is this is actually like non-trivial soliciting users so the great thing about the getting started user testing is that like almost anyone is your end user because you don't have to know anything. So it's really nice you just have to like go to focus on target demographic post messages in Slack communities where your target demo hangs out like CNCF Slack for example is a great one. I mean I was like overwhelmed with people who were like like me me me I don't know what it was it was like a time of posting or something there was like 20 people who like I want I want to like try this really badly that might not happen to you but you should be specific about the kind of users you're looking for and DM people like if you see someone in these communities that like looks like they fit your description just ask people are more than happy like people love getting like like people in CNCF Slack they're new to that Slack love getting like DMs it's like oh cool like I'm part of the community you know what I mean like and you are you sort of are and it's by far the most effective way of sourcing people by far DMing people is like in like 60 to 70% success rate because they feel special they feel like you really want to hear from them and then use verbiage like donating your time say it's the easiest way to contribute because it really is the easiest way to contribute and if possible bribe them so if you have t-shirts it's perfect plus then your logo gets out there so yeah t-shirts are key okay so now that you've got you've got users to interview you have all this stuff you're ready to go what you need to do is make sure that you have some way to record the information that is thrown at you like don't go into into these things and I'm going to go just a little bit over time sorry don't go into these things I think that you're just going to because you probably aren't so it can be a get-up issue like submit a new issue every time you hear something that someone says and then comment on that issue when you hear it again we did it's just the important thing is like make sure you can say we heard this and then we heard that again and like an easy way to basically like move like you know what I'm saying user interviews such tests are structured our structured conversations or treat them like one if someone says something that's not on your script but is really interesting take a sidebar talk to them about it but then go back to the script it's okay to venture off script but treat it like guardrails and then don't bias your participants I actually fail at this all the time make sure to ask open-ended questions not leading ones like just like ask how do you feel about this right not do you feel bad about this this is what we did so like up there you'll see some names of people and there's like sticky notes like as the background information and then all those clusters are like different like like sort of looping so sorry for the people online but like if you see like there's multiple colors in one section these people are saying the same things about that section and they're different people second to last thing listen to all plucking a feather from every passing goose follow no one absolutely it's Chinese proverb it's pretty good like you have to interview like five to six people to really get some good data but two to three can work too and then present your findings to your community we did this in the form of a PowerPoint and that'll be in the Google Drive as well and you can check out that PowerPoint and how we like rally people to like make this change and that's it so this is actually the link to the resources it's a link tree yeah one second it's a link tree there's a link down there there's also the QR code if you have any questions or if you can't get to it I think we're about to get pushed out but if we can't get to it just find me on Twitter you can ask me there or like after the talk thank you to all here's my information