 So today I want to talk to you all about the lessons that we've learned from testing our new user experience flow on wordpress.com Also known as NUX because it's easier to abbreviate things So I'm not choice. I'm a product designer automatic. So I'm also a repress guest committer Recently I've been working on the 2017 theme which is going to be released with the wordpress 4.7 next week. So keep an eye out super exciting So over the course of the last year or so, there have been a couple teams Focusing on the new user experience flow on wordpress.com So we tend to see that as everything from like when you land on the wordpress.com landing page through sign up And then up through the building of your website So my team gravity has been working on the kind of a theme and set up parts of that So to facilitate these improvements, we've run a number of a b tests hoping to increase conversions Retention and so I'm going to talk to you about some of those tests. So this could be helpful I know there's a lot of hosts here For people who have like for managed hosts who can set up their own NUX flows and actually if you are a host I would Recommend doing that setting up a unique like sign up flow to get people started with wordpress Also anyone who has a theme or a plug-in or any sort of like software as a service I would recommend looking at your, you know, your set of flow and seeing how it is So often during user user testing of sign up we'd hear people say when they get to the theme step I just want a dog site Can this can this theme make a dog site? But since theme screenshots are like really generic you you can't really like see and know what it does But really any theme can be used for any kind of site when it comes down to it if you push it enough So we thought why not ask people what kind of site they want to make and then change the theme screenshot to use related imagery So enter dynamic screenshots Dynamic screenshots props props people to enter a search term during the themes step a sign up And then it switches out the images in the demo to help people Visualize like what their site could look like So this is the first version that we AB tested And user tested so you enter a keyword and the images inside the theme preview update to match what you're entering So our hypothesis was that by allowing people to choose images and like kind of personalize their theme early They'll feel more invested in the process They'll understand oh like any theme here I can use for my site awesome, and then they'll be more likely to complete sign up And it would increase our overall sign-up conversions Yeah, so it didn't work out So we actually realized really quickly that we'd run a buggy test so we had to stop it early so we could fix all the bugs We also user tested it and found some issues that were pretty bad So there's like a noticeable flash of unstyled Content when the user started typing so like looked really bad The images were very slow. So it was just like a really unfortunate sluggy sluggish experience The search bar was easy to miss because the images were so visual that like you looked at the images and ignored the search bar Also, we were using the flicker API to get CC0 photos and they the ones that we were getting back just weren't very good So we fixed the bugs We made design and performance improvements curated the images for some top keywords and then we relaunched a new test Which looked a little bit better But it still didn't do well. Yeah Purchasing inside up and retention post sign up both went down compared to the control So, you know after looking at it, you know, we kind of figured out it's just a hard concept to understand People didn't really know what to enter into the search field. They'd enter like sentences or There was they just would put a lot of stuff Also, the search bar was still easier to miss even though it was larger So it was also hard to implement well with the available image APIs that we had So when we did our initial testing neither unsplash nor pexels, which are both like CC0 image repositories They hadn't released an API yet So it was just a resounding no, you know, we thought that maybe the idea was okay But we could get better results by doing more curation So what do we learn from this test? First off people will write literally anything into an input You might want to consider constraining it a little bit Also curation is king. If you want to guarantee good image results like you had to curate them yourself Also, make sure your tests aren't buggy And if you user test before you do your a b test you can make improvements before you push it out live Around the same time we had another team at automatic working on adding verticals to sign up So when you sign up you select a category of blog or site you're trying to make so like business arts and entertainment fitness, etc So we tested two versions one that just asked about macro categories and then a second one which asked a second question about micro categories So by gathering this data, we hope to learn more about our users and then use that to tailor things further on So our goal was less about like increasing revenue or retention and more making sure that we didn't decrease Revenue or retention while we gathered this information And we did yeah, it was great. We learned a lot about our users It's been really helpful for planning additional Testing to our next flow So we were also able to confirm some suspicions such as folks signing up for business sites or more likely to pay for upgrades Best yet both of these tests actually increased our sign-up conversion We were like well as long as they don't dip too bad, but like they actually improved it So the one page one improved it a little bit and then the two want two page one actually improved it a little bit more So we launched the two-page survey to our default flow and then we've been iterating on it ever since So recently another team at automatic has been trying different layouts and vertical configurations The user tested several different prototypes, which is a great idea by the way user testing your prototypes Before settling on this design to AB test And it actually performed much better than the previous two-step version So both the qualitative and the quantitative data pointed towards this version being just a total success So it was launched into the default sign-up flow So in a subsequent AB test we showed people who put themselves in the business vertical Business-oriented themes. So before we had just had like nine nine top themes and now we were like here's our nine top business themes So we saw a big increase in sign-up conversion And also in purchases during the first week of sign-up But we also unfortunately saw a little bit of a decrease in retention during the first week after sign-up So we weren't really positive why but we had some guesses like maybe once they got through sign-up Setting up a business site was actually really hard So we ran another test expanding this to every single vertical that we offered. So when you pick any vertical You'll see themes that we think are more relevant to you Yeah, so not very good. We saw a slight increase in sign-up conversions, but we still had that drop in retention Um, it wasn't dramatically worse, but it just meant maybe meant that we did a poor job of tailoring themes Since it wasn't that great We haven't launched it to our default flow and we're going to continue to iterate on it and see if there's something Better that we can do So, you know from these from these tests we learned that trying out one version wasn't enough You really have to keep iterating on your ideas and continuously improve them So user testing your prototypes can be a great way to find the best version to then a b test We also didn't really get a lot of answers to why like things have gone up things have gone down. Why? So we need to find better ways of actually asking our users like Why why didn't this work for you? So this year we also tested a new sign-up step that asks users what kind of homepage they want We called it triforce because there are three buckets and where's all the nerds So we identified the three most common homepage patterns in wordpress.com themes So it's a traditional blog with like a column of posts site with a static homepage you like business sites and then a homepage with a grid of Items like a portfolio or a magazine So by filtering new users towards themes that we thought were more relevant to them and having them think about the design of their site Before they got to theme selection. We hope to increase sign-up conversion. So like kind of a continuation of the previous test So, you know, there wasn't a significant improvement at first, but it was enough to tell us to keep doing this So it also came with some additional data insights. We were surprised to see that roughly half of people who signed up We're actually here to make a site rather than a blog And uh better yet This feature is actually more impactful now after another thing that came up So we really learned that all test all testing is valuable when we did this Knowing just how many folks were signing up for sites Which are much harder to set up in wordpress as a whole than blogs actually helped us reprioritize another test we had coming up Which is head start which is by far by far the most important feature that we've launched this year So it's a plugin that automatically sets up a wordpress.com site to look like the the demo that you see like in the theme screenshot and so by We we found that Having people manually like set up their sites to look like the demo is what people want the most and also The biggest pain point and customization for us For both our users and our support folks have told us this time and time again So you can see head start in action in this video. So I select a vertical And then a home page layout then a theme Uh, then I go through some additional steps signing up Cool and so I finally get to my site and it's set up to match the demo I saw during theme selection like magic So it's like there you go it's site So our hypothesis was that it's going to be easier to edit something Than it is to create it from scratch. So by setting up your site for you Including demo content will increase your likelihood of sticking around Yeah, so we uh, it was awesome. We saw a decent increase in the number of people who came back At least once in the first week after setup I'm pretty sure we saw a decrease in happiness like support Requests dealing with setup So it was kind of a no brainer for us. We were like, yeah, we you know, we're launching this So we've since expanded it to cover almost all free themes And we're actually working on expanding it to cover premium themes as well And when we combined try for us with head start We actually saw a huge increase in the number of people who came back at least once in the first week after sign up So after launching it, uh, we kept getting a lot of requests from our support folks To create a way for them to head start a site that had already been edited So I guess a lot of people start their site. Um, they edit it a little bit. They're like, oh my god This is a disaster. I need to start from scratch and they've no way of doing that now So what they would do previously is make a new site and that's just kind of a mess So our happiness engineers were like, can you build us a tool to head start a site? And so we were like, sure and we did So it's been pretty good and we're thinking of maybe expanding it to all users eventually next year, but Maybe I don't know So yeah, we confirmed that editing is way easier than starting from scratch And if you do the the hard parts for people, they're going to come back And actually use their new sites Also, sometimes people get into building sites and they mess them up and they want to start from scratch So it's good to provide people a way to like reset So some of you may actually already be familiar with this project because it's launching in uh four seven next week So direct manipulation is the idea of being able to click something and then edit in place So we wanted to test out a light version of this so not completely direct manipulation, but kind of So since not everything you see in the customizer preview is actually editable yet We started by adding icons to the icons to the items that you could edit So when you click an icon, uh, it brings you into that section of the customizer and you can make edits We launched our first version with buttons that were at edit and then our second iteration We changed it to the button icons that you see here So we hoped that by making it obvious what you you know, what you could edit and then Bring you to where you could edit it. You'd be more likely to actually, um Successfully customize your site and then customizer saves would increase as a result Yeah, so we found quickly in user testing that people did find use and uh Like succeeded to great success. Uh our ab test data backed this up So customizer saves almost doubled. Uh, it was a pretty clear success for us Yeah, um, you know, we kind of knew people were clicking on things in the customizer trying to edit them Because that's kind of what you you would expect this day is like to be able to do that But now there's like actually a way for them to do that. Um, and It just it's made customization way easier. I use it all the time now for site setup Uh, and also when it comes down to it, this wasn't a huge project You know, it took us a couple weeks and and that couple weeks that we built it We almost doubled customizer saves. Uh, so combined with head start, especially it was a huge win Uh, so when it comes down to it This feature is a band-aid on the underlying issue of the lack of true direct manipulation and front end editing Wordpress, uh, and that's okay because it's still an improvement and it's a step in the right direction And it's coming to core. So that's great. So our last test, uh, was a feature designed by um, fellow automatic designer Davide Kassali that my team coded and tested So in trampoline when you sign up for new wordpress.com account and you make a site We tested out a new flow where you'd be dropped onto your home page And then your admin menu would pop out and you'd you'd be like, hey click on this stuff So by dropping users onto their site and then telling them where to go next We hope to increase retention within a user's first week Uh, and it did, you know, people definitely came back a lot more. It was actually one of the more successful tests that we ran So we did a second iteration pointing them to the customizer like hey go customize your site and people went to customize their site And people who did that were more likely to come back after their first week So we've experimented with a bunch of places to drop users in right after sign up their dashboard their stat page the reader on wordpress.com But it seems so far the most effective place is your site itself But if you do that, you shouldn't just abandon them and be like cool. Here's your site. Bye. You should tell them what to do next So amongst all this testing, what did we see consistently? Well, like Sean Andrews says here Uh, straight up asking people what they want Helps us help them set up the sites they like and want to use And then showing them how to make additional edits actually increases their likelihood of coming back So I'd encourage anyone selling products in the wordpress space or services To ab test and user test everything you release So there's a lot you can learn through testing to improve your onboarding and overall user experience Thank you