 So we're gonna start the presentation now. So thank you everybody for coming to our talk. Today we're gonna be talking to you about a case study on how designers can shape better documentation. So we call the study re-imagining Knative. So a little bit about who we are. My name's Zainab and I am currently a UX design researcher at OCAD University from Toronto, Canada. I'm also one of the design leads for the new Knative UX working groups. This is a fairly new group. We began in September and we've done a lot of exciting projects that we're excited to talk to you guys about today. I also have a background in human-computer interactions and I've been involved in design education at OCAD for the past two years. All right, hello everyone. I'm Marianna Mejia. I'm also a UX researcher at the same lab Zainab is at at OCAD University. I'm the other design lead at the UX working group and I'm finishing my final semester in the industrial design program at OCAD. And some of my interests include accessible design, human-computer interaction and authoring tool design. So before we get into the guts of our presentation, we wanted to give you a roadmap and give you an idea of the questions that you should have answered by the end of these slides. So the three main questions are what, why and how. So first of all, we're gonna cover what is Knative and just so that you know what the documentation we are restructuring is. Next we're gonna give you an overview of what UX design is. Since I'm assuming most of you do not have a design background and we're gonna be covering some buzzword topics like for example, how to make something more intuitive or user-friendly. We're also gonna be covering why. So why might you want to explore UX strategies for your own projects and why you might wanna think about involving designers in your open-source tools? Lastly, we want you to consider how. So how can you actually implement these UX strategies in your own projects? So an example we're gonna show you is card sorting. And lastly, how can you actually go about finding designers for your open-source projects? Right, so first getting into what is Knative. Knative is an automation layer that you can use with Kubernetes to deploy native Kubernetes services. So it has three components. The first component is functions. So this is a simple programming model that allows you to easily create and deploy your own event-driven functions. We also have, sorry, we also have serving which allows you to deploy and run containers. And lastly we have eventing which enables different events to trigger container-based services. So three primitives or components. And so why might you want to use Knative? So the benefit of using Knative is that it allows you to automate things. So there's less of a need. So if you were to use Kubernetes by itself, you'd have to set up the complex infrastructure. And by using Knative, you as a developer can focus more on your own code. It also has less resource consumption. So Knative is able to start and stop instances automatically as needed. And lastly, there is auto-scaling. So you only pay for the compute time that you need so it is gonna be cheaper. Okay, so we've covered the tool that we're talking about today. And now we want to talk about why open source has a need for user-centered design. So in many cases, the developers are not the only users of the tools. And while in the past, the groups of the developers and the users were the same group. But as software has developed and software has taken over many industries, as you guys know, the end users of open source are not only experts. And we want to be considering this gap here. The users that are not ex... Oh, there's no cursor. There's this circle on the left, the bigger circle, which shows that not all developers are users. Okay, so now we're gonna give an overview of what UX design is. So our goal as designers is to both understand and improve the experience between a user and a product or service. So the big question you wanna answer is how can we make the entire experience more intuitive or user-friendly? And I also wanna point out that UX design is an umbrella term that covers a variety of topics. So a lot of people might hear UX design and think it's only about the interface, but it's actually much deeper than that. We also do things like how people navigate a tool, how you can structure and organize your information. We consider things like accessibility of tools, and we also do things like user research and user testing to ensure that all the features are actually what people want. Okay, so going back to this word that I keep using and that you've probably heard before, so what makes something intuitive? There's a typo on the slide, sorry about that. So how can we actually design our tools to be more intuitive or user-friendly? There's basically two aspects to this. The first aspect is leveraging a user's prior knowledge. So I'll give an example of a command line interface. So for example, you have a user and they use a command line interface where the components are in a specific order. And say you have a new product and the components of the command are in a different order, the user is not gonna be very happy because this goes against their prior knowledge. So you want to be designing tools in a way that is familiar to what the user has used before and you want to follow certain conventions that are common. And so in order to figure out what's common, you have to kind of know who the user is and what their experience is, right? So that's where the user research part comes in. And the second part is considering human instinct or common sense. So an example of this in visual design is where if you have an important piece of information like the title, you should be using a larger font or using a contrasting text as the eye will intuitively be drawn to this and perceive this as the most important piece of information. And like I mentioned, these design decisions often need to be researched and need to be tested to ensure that your product is actually following these two principles. Okay, so now we're gonna get into what the design process is. So the design process has five stages and the first stage is to empathize with the user. So you wanna start off with a broad understanding of the user's needs. So while you can hypothesize what the user's needs might be, you don't really know unless you go out and ask people. So this is where you're doing things like observations and surveys. The second stage is to define the problem. So once you've collected all your data and information, you can now begin to narrow down and define the problem space. And once you've figured out what the precise problem is, then you can begin to ideate solutions. So sometimes we as engineers tend to jump to the ideate step, but we wanna make sure we're really putting emphasis on the first two steps, which are more of induction. So in the ideate phase actually, in our Knative Working Group, we end up working a lot with a technical team to come up with potential solutions to the problems we've identified. And after you've come up with potential solutions, you can begin to prototype solutions. And this is where you might start using tools like Figma to design a potential solution, and then you can start getting into developer handoff. And once you've done all that, you can start testing things. So you can show people what you've made and get some user feedback on that. And that is stage five. Okay, so a bit of an overview of how designers can help open source. Basically, we want our tools to be benefiting as many people as possible. So I have this diagram here showing the positive feedback loop of why UX design is important. So first, if you have more emphasis on design, you're gonna end up with tools that are easier to access. Tools that are easier to access result in more people using your tools. And the more users you have, the more likely you are to have more contributors. So it just is a cycle that we wanna keep continuing, right? And here are some more ways that designers have been beneficial and K-native. So now more of our decisions are user-driven, which means that the new features and tools that we add follow, or hopefully, are following user demand. So the features that we develop are actually gonna be used by people. Next, by having better information design, we're able to have more efficient onboarding for both contributors and new users. So an example of information design is our documentation restructuring, which Mariana will be going over in a second. And lastly, we wanna focus on things like product improvement for existing products. So for example, for our command line interface, we were looking at ways to make it more intuitive or user-friendly using the principles I described previously. So here's where we are analyzing and improving our existing products. Okay, so hopefully by now, you are thinking about how you might be able to involve designers in your own products, but there are still a couple challenges. So while developers may be aware that better design could be beneficial, they may not always know the specific strategies that are helpful for their own projects, and they may not always know which ones to implement and how to implement them. And lastly, there are a lot of terms that they may not be able to articulate to designers. So we hope that during the rest of this presentation, we're able to address some of these concerns. All right, so that brings us to our specific case study that we're talking about today, which is have you ever been stuck in documentation health? Let's say you're trying to look for a specific feature within pages and pages and documentation, and you can't seem to find it. Or maybe you yourself are a maintainer of a specific open source project, and you develop the feature and they tell you, okay, now put it in the documentation, and you're like, okay, but where does that fit? This exercise is kind of meant to help you figure those kinds of questions out. So we'll start by contextualizing a bit here. This is the candidate of website, specifically under the eventing page, and we were approached by maintainers from the eventing working group at Knative to kind of examine this substructure that's towards the left, and see if there were any usability issues or UX improvements that we could make to it. This is just highlighting the submenu that I was referring to. Okay, so this is actually a difficulty with information architecture. Over here, you can see towards the left column, there's like the unexpanded version of the submenu, and then to the right, there's the expanded version. There's a lot of categories underneath this and a lot to go through. And so we want to figure out how to structure this information. So what is information architecture? It's kind of like a subdiscipline within UX design where you're trying to present information to the user in a way that's understandable, easy to find, and that makes sense on like an intuitive level, and it categorizes and presents relevant information. So for this issue that we have presented, what specific UX research strategies are there? So there's a couple of research strategies within UX design, some of which you may have heard of or done yourself. There's things like surveys and interviews, but there's also more specific ones like co-design where you're designing with your end users or you're doing digital ethnography, which comes out of anthropology and it can be thought of as like noting people's behaviors, say on like online forums, for example. You can also do things like diary studies where you ask a user to maybe interact with your design and write down their own notes on how they feel during the interaction and analyze that. But luckily there is a specific UX research strategy that is meant for information architecture. This is called card sorting and it really helps you understand a mental model of how your users understand the information on your website or your specific project. So it helps you answer questions like how do people categorize different pieces of information? What pieces of information are related in their minds? What's more important, what's least important? Things of the sort. So this is just a very quick example on how to apply card sorting just to like kind of onboard you to the idea. Let's say you're a designer at Microsoft Teams. The Teams interface has not been invented. You know that you want something like time since beginning of the meeting, a leave button, a mute toggle button and maybe a video toggle button. But that's really all you know. So you can write all of these down into sticky notes or cards and give them to end users and see how they categorize things. So you might get a result like they might group together the mute toggle button and the video toggle button and they might place an emphasis on the leave button and less of an emphasis on the time since beginning of the meeting. And they might even tell you something like you're missing a chat feature. So you actually, once you analyze those results, you actually get something that ends up looking like the navigation bar on the Teams interface. So you might find that people place more emphasis on the leave button so you make it red, bold, easy to find. Less of an emphasis on the time since beginning of the meeting. So you put it off to the left in less bold lettering and then you put together the camera toggle and the mute toggle because they're conceptually related. So that's how you might translate that into a graphical user interface. So going back to our original issue of confusing documentation, how did we even get here? So as you might already know, you might develop a feature within the open source project and not know where to put it. So this is a common issue that also happens in Knative where people maybe categorize things in a way where most people wouldn't think to find them. And the end result is that it's hard for contributors to find the specific feature. It's hard for end users and it's hard for maintainers to kind of keep up with the whole documentation and its structure. So we were looking at the following questions. What subtle changes can we make to this left side menu so that it's more navigable? We also had some more specific questions surrounding two topics of developer topics and administrator topics, which are kind of like sandwiched in the middle here. We wanted to find out whether or not people actually use them to understand and like find the specific thing they're looking for within the Knative documentation. And we also wanted to find what other main categories do people use, like what should be up here and what could we do without? So actually at the last KubeCon, the wonderful people at Knative that volunteered to be at the booth conducted this study and yeah, we had a total of 25 participants. This is a bit of what some of the boards looked like. It's just like a poster board with sticky notes and the information found on the left submenu kind of like abstracted out of the interface. And over here, we have like a funny thing that I like to call the corner of shame, which is pointed out at the bottom right hand corner. It's just like a little corner where people can put sticky notes but they don't think are relevant. And I'll get into a bit more of why that's useful. So how do you analyze card sorting? We did more of a quantitative approach to the card sorting data. So we were trying to look at frequency of like what were like, let's say, the frequency of administrator topics and how much it was used to categorize other groups and what topics were frequently grouped together. This is just a bit of what the analysis looked like in the images. Okay, so back to our original questions. Do people really use a distinction between administrator and developer topics? In our specific use case, the answer was yes. It was actually the two top used categories that people use to make sense of what was in the left submenu followed by observability and resources and were there any categories that people could do without? The answer is yes, they actually erased something called, it's a bit nested, but it's called the sugar controller. And that's interesting because we hypothesized it was due to a lack of information. Like people don't necessarily know what a sugar controller is. Therefore they didn't know how to categorize it and therefore they put it in the corner of shame. Then we had some emerging questions. We had other emerging questions besides this one, but maybe this one was the most important. Were there specific pieces of information that people categorized under administrator or developer topics? And the answer was there was no, I guess, significant overlap between the different boards and how people categorized them, but there were somewhat consistent broad themes. So under administrator topics, you could find things like collecting, observing, configuring. Meanwhile, under developer topics, you could find things like troubleshooting. And what were the final results for? Like what changes we made to the actual interface? So to make a long story short, we renamed some topics. We moved to other topics around. So for example, we moved up the resources and the developer topics and administrator topics because we found those were relevant. We made some visual changes. And because of the sugar controller, we thought to add frequently asked questions in order to fill in that kind of educational gap that we found. So where else could you use card sorting? Is it only okay for documentation? And the answer is no. It's really good for anything that has information in it, which is, I would dare to say all interfaces. So you can use it for the navigation bar on your website. You can use it to figure out the graphical user interface of your project. You can even use it to structure a command line to understand what structure for the command lines and what order makes most sense to people. And really any other information architecture issues. So hopefully you're interested in conducting your own card sorting exercise. And how do you do that? So this is a bit biased towards how we did it. There's actually many different types of card sorting exercises, which I won't bore you with today, but I would be happy to talk about it some other time. But yeah, you just have to get the supplies. You can do this digitally with platforms that kind of imitate whiteboard layouts or sticky notes. This includes things like Miro, Mural, and Fig Jam, if you guys have ever used those platforms. But for in-person exercises, you just need the poster board and the sticky notes. And for the second step, you just write down whatever categories or pieces of information you want to understand onto the sticky notes. And this is actually a very important part. When you're placing those sticky notes on the board, be sure to randomly put them there. Otherwise, you're kind of biasing the way you want people to place the sticky notes. You don't wanna indicate how you think it should go. And as an option, you can add blank sticky notes so that if people think there's a conceptual gap missing, for example, that chat feature that we were talking about in Microsoft Teams, you know what you're missing in your interface. I would also highly encourage, if you can, ask questions as to why they think it's missing. And then as an option, you can add the corner of shame. It also provides some very useful information as to what people think is extra information or maybe they don't know what it is. Again, it's very interesting pieces of data. And then obviously you contact participants. It can be a bit tricky to source participants sometimes, but yeah, once you do, be sure to follow ethical guidelines, gain their consent, things of the sort. And I would also highly recommend getting a video if you can of the participant sorting out the cards so that you can kind of see their rationale. Like what decisions did they go back on? What did they have trouble placing? That can tell you a lot of things. And lastly, analyze. It's kind of easier said than done and you can do it in a variety of ways. But yeah, look at what hierarchies exist. What do people place as categories the most? What patterns are there in the data? And be sure to be responsible with the data collected and if you're sharing findings, please anonymize the findings unless you have explicit consent to not do so. And then try to analyze those results. This is where having a designer really helps. Try to translate those results from the card sorting into whatever interface that you're making. So you can place like highly relevant things as more visually important. So maybe add some more contrast, make it bolder. For less relevant pieces of information, you can minimize them, push them off to the side. Things of the sort. Consider adding new information where people thought it was missing. And then lastly, share the findings. Obviously it's open source so it's always great to share with the broader community and get some feedback from subject matter experts in your specific project and implement it. Okay, so now we're gonna be covering some of the challenges that we faced during this card sorting exercise. So the first challenge was to figure out how to actually find creative users to interview. So our solution was to do it at KubeCon cause a lot of people come here. So this was at the last KubeCon and although we did get 25 participants, we're thinking that in the future it might be good to also have a digital version of this so that people that are international or people that aren't present at KubeCon physically could also be able to participate. So that's one way that we can actually improve our sample size. And one of the other issues that we had was that the volunteers that were running the study and also the participants did not have a prior example of how this type of study is run. So it would have been good to inform the volunteers about guiding questions or tell them to write down more notes as the study was being done. And this leads us to the third issue which was a lack of context in analyzing. So me and Mariana were analyzing the study results from the photos. But unfortunately, because it was already anonymized we did not have a way to ask any follow-up questions to get a further understanding of people's rationale. So like we mentioned in the future it would be good to either have people taking notes or having some kind of video or audio recording of the actual process of sorting and having people ask questions as the study was being conducted. Okay, and here are some of the general challenges we're trying to figure out in our UX working group at Knative. So in terms of onboarding new design contributors there is like challenges in like technical knowledge gaps. So we found that having synchronous meeting has really, synchronous meetings has really helped us especially like in the cloud development space there are a lot of technical concepts. So having a lot of communication between developers and designers has been really helpful. The second major challenge that we're still trying to figure out is in terms of tracking non-code contributions on GitHub. So this is a pretty major challenge and we're still figuring out the procedure for this. Something that we've been doing so far is that whenever there is like a new design issue so we'll open an issue on GitHub and then the designer once they've come up with their design will open a pull request and within that pull request we'll either have a PNG image of the design or a link to a Figma file and then people can approve that pull request when that design change has been approved. So that's what we're doing currently but we're still like exploring other ways to do all this. And lastly we are still doing some exploration about potential design tools to use. So like I mentioned Figma it's a pretty common tool in the design space but unfortunately it does have a paid license so we're trying to figure out ways to give our open source contributors access to tools that are free so that they don't have to pay money to do open source contribution. Okay, so you also might be wondering why designers would even wanna participate in your open source projects. So there actually are a whole bunch of reasons. The first reason is that designers really enjoy seeing their changes implemented in an actual deployed product and it's something really great that they can add to their design portfolio. A lot of people are also interested in improving their technical knowledge so that's another benefit. And lastly they may actually wanna help make software tools more accessible and this way more people get to have a say in the features and improvements for the tools. And lastly open source is an amazing community and we've had a really good experience with it so far. Okay, so hopefully by this point you're like, oh wow I wanna start my own UX research team and my own project. So here are some ways that you can go about doing that. The first is that you could find a organization with an existing designer demographic to partner with so you could find a school or a research lab or even a local community meetup group and you can start talking to designers there. The next step is something that we are hoping to look into ourselves is to start hosting more design events. So for example, you could have a designathon and get different people from the design community aware of the problems in open source and get them involved. And yeah, from our experience we definitely recommend having regular synchronous meetings, especially for the purpose of developing a shared vocabulary and learning what tools each group is using. And yeah, so if you do end up onboarding some designers you might need to give them some guidance on how to navigate GitHub. And here are some of our ongoing and future projects in our working group. So the one that a lot of people are very excited about is our mascot, okay, thanks, is our mascot. So this is our mascot Quack, which we had the pleasure to redesign recently. So that's him. And yeah, besides that, we also have been doing some website redesign. So the version on the left is the old version and the version on the right is a new proposed version that we're still implementing and deploying. And besides that, we're also working on a mobile version of the Knative website. And we're also hopefully gonna be developing or starting development on a Graphic User Interface for Knative pretty soon. And lastly, we have a contributor experience and retention rate study that we are currently doing as well. Okay, so I wanna give a huge thank you to our team. So both our team at OCAD from the Perceptual Artifacts Lab and also everybody from Knative who's helped us with all of our projects and also these lovely people. And we also wanna give a call to action. So if you have ever used Knative or have any ideas for new features, we would love to hear from you. So Quack, the Knative Doug invites you to participate in our UX study. The link is here. So I'll come back to the slide in just a second. I just wanted to go to the next slide. So and lastly, I wanna invite you guys to get in touch with us. So you can connect with us either on LinkedIn or via email. And I'm also gonna be at the Knative booth after this presentation. So we'd love to talk to you about getting designers involved in your own projects as well. Okay, and these are our sources. So yeah, thank you. Any questions or comments? Hello, yeah, okay. Thank you for the talk. It was very nice. I was just wondering when you ask people at the booth to hold these kind of car sorting exercises, you say to educate them about guided, not using guided questions. Can you give examples of what good questions would be? Oh, yes, that's a great question. So a good example might be like, oh, don't you think these two should be categorized together or maybe like almost like incriminating questions? Like it also has to do with tone. Oh, really? Like do you think those two should go together? Like that kind of thing might make users, or participants second-guess themselves, and then maybe bias towards changing their answers. So be careful. Also, I wanna make sure, so were you asking about biased questions or questions to help with research, like useful questions? Yeah, okay. So Mariana gave an example of a biased approach to asking questions, and maybe you wanna ask kind of open-ended questions. So maybe if you have two categories that the group together is like, oh, what do these two categories have in common? Or where have you seen these two categories before? So just kind of like approaching it with curiosity and keeping it open-ended. And yet what you don't wanna do is like, don't you think that this one should be over there? Like you don't wanna ask those kinds of questions. Oh yeah, does that answer your question? Yeah, okay, great. All right, thank you. Did you have a chance to measure the impact of your changes in the menu? That is a great question. We definitely wanna look at like a user test maybe to see how effective, like that's the last step in the design process. We haven't gotten around to that portion, but absolutely we do want to do that. So the answer is no, not yet. And something that we might actually be doing is that for people that we are asking to participate in this study, depending on who signs up, we might ask people to like take a look at the new documentation and comment on whether it's more or less helpful than the old one. So that's the one way that we might approach the testing part of the design process. Do you have any comments on using, for example, some kind of AI into the documentation for users? If maybe I'm not certain I'm not like looking for a specific word or something, just like have an overall question about this documentation. And sometimes the search engine is not the best way sometimes to search for things that I'm not certain how to ask. And maybe I've seen some other places where you just like ask the AI bot and the AI bot looks around the documentation and give you an answer. Yeah, we haven't looked into any like chat bot style features for any, I mean, as far as we know for like specific to documentation, question chat bot stuff, but that could be if there's enough interest that could be a potential future project. Do you have any thoughts on this? Yeah, I think that's actually a great suggestion and definitely something we could look into, but it didn't necessarily come up during the study. I guess because we were asking like on a very specific part of the documentation and like card sorting, maybe during an interview it would have come up, but yeah, absolutely. I could see a lot of benefit in that. I think probably something that is more of a short term goal would be to have like a search bar features a good search of certain words in the documentation as a whole. So we could implement that. Yeah, thank you for the great presentation. So for your example for K-Native, the menu was already there and categories were already there, but let's say we're starting from scratch. What would be your idea of defining categories or how many categories should we even have in the menu? Yeah, any thoughts on that? Yeah, that's a great question. Okay, so there's a couple of things or Zaydem, do you want to tackle the- You need to go first. Okay, yeah, so this is maybe where the different types of card sorting could be helpful and also some like old cognitive science studies suggest for example, seven pieces of information so you could maybe try to aim for seven high level categories. Seven, I think it's like plus two or minus two. And you can start off with that and you could do a type of card sorting which is a lot more open. So for example, you provide the users with the seven categories that you're thinking of and then you ask them to fill it out for you. That one can be a bit trickier and maybe demands more time than this kind of card sorting, but yeah, that may be helpful. I think also for that situation, your users would have to be more familiar with the domain as well. Like maybe they've used other tools or they have like expertise. So you could definitely, I'd probably start with like more of like a, if you're like a non-technical person doing kind of like an open-ended discussion-based interview, be like, oh, like what kinds of information do you typically put in documentation? And then like once you have like a conversation, you could pick out keywords from that conversation as well. And that could go on to your cards as well. Thank you guys so much.