 Hey everyone, we've just approached the hours. I'd like to show you what UX research we've been up to recently. So earlier this week, we concluded some testing on the merge request page. We decided to work on the merge request page following a comment on Hacker News, which mentioned that GitLab feels extremely crowded and has a general lack of polish and Annabelle in the front-end team pointed out that the merge request page felt particularly overcrowded. So the length of a merge request description varies from one merge request to another. There's very little consistency. So subsequently this causes the merge request widget and the discussion commits pipelines and changes tabs to move around on the page. And it's a very frustrating experience like scrolling up and down trying to locate these elements. So Dimitri started working on the design to improve the merge request page and there were four main changes we wanted to test with users. So they were displaying the contents of the merge request widget inside of a tab. So the widget would no longer exist on the page. Making this tab open by default when somebody views a merge request. So just to remind everyone, the discussion tab is currently open by default at the moment. The branch name was redesigned and given a different visual style. So with the assumption that it would be easier to locate. And finally, longer descriptions were now truncated. So users could still expand and reveal this description if they wanted to. But it meant the tabs would always be in a fixed location regardless of the length of the merge request description. So as we wanted to know whether the changes we were proposing to make were an improvement to the existing experience, we also needed to take a baseline measurement of the existing experience. So that just means we wanted to be able to basically quantify how much of an improvement that we've made and equally so if the changes weren't an improvement, we wanted to know how far away we were from achieving a better experience. So this requires two prototypes. So one of the proposed new design and one of our existing merge request page. Now, you might be wondering why we mocked up a prototype of the existing experience. It seems quite a strange thing to do, but it's important that when users are comparing two designs that the testing environments feel the same. So there's a possibility that we could skew users opinions by showing them the existing experience live on GitLab. So because that would obviously feel more complete and whole to a user than a prototype would. So Dmitri built two prototypes for testing prototype pay, which was the current merge request page and prototype B, which contain the proposed changes. We tested with the 11 users to split with six GitLab users who are familiar with merge requests and CI and five non-GitLab users, but who are familiar with merge requests in a competitor's tool, such as GitHub or Bitbucket. And we designed tasks that deliberately moved users around the merge request page and which forced them to scroll and to try and locate the tabs and so on. Now, we couldn't ask users to complete the same set of tasks on both prototypes because obviously after interacting with one prototype, they start becoming used to where they might find things on the page. And that increases the likelihood of them completing tasks on the second prototype much quicker and that skews the data we're collecting. So what we actually did was give them one of the prototypes to complete the task on first, and then we show them the second prototype and then just let them freely explore it and ask them things like what they thought the differences were and which they preferred. For each task recorded, whether they were able to successfully complete it. And we also track quantitative metrics such as how long it took them to complete the task and how easy it was to find what they were looking for. So we let them rate each task out of five. And from there, we were able to work out an average satisfaction score and time for each of the prototypes. Now, there's a very famous saying in UX research and that is you need to watch what users do, not just what they say. So the majority of our users told us they preferred prototype B, which was the design that had the post changes to the existing page. However, sometimes users actions and the quantitative data we captured didn't actually support that. So as I mentioned earlier, we had four changes that we wanted to test. So we had to design the research in a way that if prototype B didn't do well in testing, we could understand why. So was it one change or was it a combination of two or three of the changes? So I'll quickly run through the results for each of the changes. The branch name tested really well. It had a higher success rate and satisfaction score. And users were also much faster to locate the branch details on prototype B. So they were over twice as quick as prototype A. So that tells us that the branch name could be safely implemented. The description expansions, there was a bit of a split in user opinion. And generally, this seemed related to the type of workflow a user had. So users who work at pace, opening and closing merge requests within a few days wanted the description to be truncated. And that's because they have a good idea about what the merge request was about. And they didn't typically need to reread the description. And users who reviewed merge requests, which span maybe weeks or even months, especially those in a research field, wanted the full merge request description visible at all times, as they would often refer to it when revisiting the merge requests. So manually, like having to expand the description every time they revisit that merge request is really obstructed to their workflow. So in summary, we have a difference in opinion depending upon a user's workflow. So we might want to think a bit more carefully about what we could do about this before we go ahead and implement this change. The merge request status tab open as default didn't work well at all. Before we showed users either of the prototypes, we actually asked them what information they would look for first when reviewing a merge request. And the top answer was diffs, as users want to understand what changes have been made to the merge requests. I also compared how many users stated items that could be found within the merge status tab compared to the discussion tab. And the discussion tab was more popular since users wanted to know what comments, if any, have been made by their colleagues, whilst they've not been looking at the merge request. In addition to this, we also saw users who are used to GitLab's existing experience accidentally visit the merge status tab when trying to view discussions which caused some frustration. And we also witnessed users change to the discussion tab and then fail to switch back to the merge status tab. So they subsequently failed later tasks in the testing. And it was almost as if they'd forgot about the new tabs' existence as their view by now having the discussion tab open mimics the existing experience. So the users who did, and it was only one or two users that did prefer the merge status tab open as default, were typically working solo. So they were working on like hobby projects or they were in a very small team. And therefore they didn't actually utilize the discussion tab. There was one user in particular who said, my team all works in the same office and we work very closely together. So we don't tend to have discussions because we can just turn and tap each other on the shoulder. So hence their preference for the merge status tab to be default instead. So in summary, there were some users that were just so used to the existing order of the tabs that they weren't able to switch their mental mode during the testing. So we know that if we rolled out this change that it would take users a while to get used to it and it could cause some frustration to users who typically view the discussion tab immediately. Now, the merge status tab itself, well, despite users preferring prototype B, I personally feel it needs a bit more work. And now this is truly my personal opinion. Derita is on vacation at the moment, so we've not had a chance to catch up. So I was just speaking on my own behalf here. I'm really pleased that we did test the new tab as I feel it's given us a lot of insight that perhaps we wouldn't have got from the just testing the existing experience, but to quickly summarize why I feel the tab needs more work. The first thing is that it took users twice as long to locate the merge CTA on prototype B, and both the success and satisfaction rates were slightly lower than prototype A as well. And we saw a lot of users and this is probably just over half of all users looking for the merge CTA near the comment box within the discussion tab. And the reason for this is because it's where it's located on GitHub. So users are bringing a common mental pattern from a competitor tool into our own tool. Now, I don't believe we should copy everything our competitors do, but when many users instinctively follow a path to find something, it does raise the question of whether we should have a similar implementation. And secondly, no users look for the issue number within the widget or the new merge status tab during testing. They found it actually in the description. Yet multiple users said it was really important to them that the issue number was visible at all times and easy to find. So obviously based on what we witnessed during testing, that's going to depend on how well someone has written and structured their merge request description. If users instinct is not to look within the widget or the new merge status tab, there's still a risk that the issue number could become lost. So should the issue number reside outside of the widget or tab as its own element on the page? The iconography using the merge status tab didn't make any sense to users. They prefer to actually click into the merge status or the pipelines tab to try and get a better understanding of what was going on. Therefore, do we need these icons? And finally, while users were quicker on prototype B to spot the status of the merge request, they were more likely to provide a more detailed status answer. So like referencing things like unresolved discussions or the pipeline status on prototype A. And is this because the information was available to them immediately? In some cases, users had already changed tabs by this test. Therefore, they would need to make an additional click back into the merge status tab to provide a more comprehensive answer. So the next step is to answer all these questions as a team. The likelihood is we will iterate on the design and put it back in front of users to test. We also discovered additional problems with the merge request page, which are actually unrelated to the changes that we were testing. I'm not going to discuss them today because there's far too many, but if you would like to read what they are, they are listed within issue 21 of the UX Research Project. So we also need to decide as a team how we'll address those problems as well. So what else is going on at the moment? Well, the navigation has gone live. I know Sarah mentioned it yesterday on the team call. We have an issue that is capturing feedback, and we've asked users to answer a few questions when they're providing us with feedback. It's very easy for people to look at something and say they don't like it. We want them to be more self-aware than that, and at least attempts to understand why they don't like something and then share that with us. That gives us more constructive feedback to work with. And tomorrow, next week, I'm actually speaking with more users. So I mentioned in my last update that I was going to look into notifications and dashboards. After reading through issues for these topics, I realised that some of the questions in the issues were actually broader in scope and related to user workflows. So notifications and dashboards will still be my primary focus as the research, but I've also asked users to share what their workflow is with me. So what does a typical day look like for them and how does it fit into their workflow? Speaking to a broad range of users, we've got some C&E customers. We've got developers, system admins, product owners, and we've got a mixture of small and large teams as well. So I'm really looking forward to kind of understanding the differences and the similarities and hopefully you can share those with you in my next update. Does anyone have any questions? I can just see Annabelle's question. Have you considered a prototype with both the merge status and the description are included in the discussion tab? I think it's been talked about before, but that is not what we kind of tested. But yeah, possibly that could be a good idea based on the results that we've got. We've made two sets of users very happy. Was there a correlation between prototype preference and being an existing GitLab user from Chris? No. It was kind of a mix. The interesting thing that I did find was that there was two non-users at GitLab who looked up both prototypes and went, I don't really see any differences between them. So it kind of shows that I suppose they were quite subtle differences to non-users, and therefore they didn't really have a preference. I think again, like I mentioned, it really depended on the kind of team that they were working in and what their kind of workflow was as to whether they preferred a certain prototype more. And this is why I think the research that is upcoming tomorrow and next week is really important because it'll give a bit more evidence to the kind of research that we've been doing where we've kind of got bits where we know it's different, but we can go a bit more in depth with it. Have we done tests and whether people care about real time? Well, someone actually did mention real time during the testing. They wanted the widget to be real time. But other than that, they haven't really had much update on that. But I think that will come in the next round of testing because we're going to be asking about notifications. So we'll be asking people like whether they used to do and also just generally, whether they like, for example, get my emails to tell them about events that are happening or whether they prefer to work through to-dos and how they kind of manage them, but also the kind of other tools that they perhaps integrated would get lab and mean. For example, chat clients like Slack, do they get notifications for them? Are they real time notifications or do they find it distracting? So we'll kind of get a better sense of that in the next wave of testing and that's something I can definitely get some answers on. Anybody else? When users talk about the merge request page being crowded, do we know which workflow they tend to use and which areas of the page matter most? We know bits from this kind of wave of testing. Some users during the testing, by all means go and listen to the videos, will give you an idea of how they actually use that page within a team setting and the kind of test they perform versus perhaps what their teammates perform on that page. I think, again, it's kind of, it needs more research into, I feel like we do a lot of workflow of things in isolation. So the merge request page is still a singular page, I suppose. Whereas what I'm hoping in the next wave of research, it will be literally everything from the start of some Wednesday to the end of some Wednesday rather than just something in isolation. So I don't know if that answers the question really, but I think that's it. So thank you very much, everyone. Enjoy the rest of your day.