 Alright everyone, this is a short session and we're going to get started. This is A.B. testing with Optimizely, hopefully you're in the right room, if not, sneak out. Alright, quick introduction, my name is Kristen Paul, I live in Santa Cruz, California, in the United States, I've been doing Drupal for a long time, in tech for a long time, and here's some contact information, my silly kids from a few years ago. Alright, some goals for today, we're going to give an overview of A.B. testing in general, learn a little bit about Optimizely, and then there'll be some resources on the slides that I'll post after the session. Alright, so to start with, what is A.B. testing? A research methodology that compares two or more versions of a product or strategy determine empirically which version is superior. Okay, what? It's a lot of words for something that's really not that complicated. So, we'll talk a little bit about how A.B., the concept of A.B. testing started, actually in the early 1900s, it started more in agriculture, and there was this idea of having two fields and having more fertilizer on one field, and see what happens. Well, that's like, they didn't call it A.B. testing back then, it was sort of the scientific method, but it was a very simple example of how to do A.B. testing. Fast forward, and that was started to be applied in the 1950s to clinical trials in medicine. So, it's a pretty standard thing, you don't normally call it A.B. test, sometimes they will say that group A and group B, but usually you have like a placebo group who's not getting anything, and then you have someone who's actually getting the medicine that they're trying to test for. Well, marketing decided this is kind of a cool thing, why don't we do this? In the 1960s and 70s, print marketing started to use this for things like postcards. So, they would send a postcard to two different kinds of postcards that were different, and maybe the phone number would be different. So, they would know which card you got and what kind of response they got. And now, more recently, since around 2000, we have A.B. testing for the web. So, for the web, the actual term A.B. tests, they attribute it to Google. It's not really clear, it really was Google, but they say the first official A.B. test was in 2000. So, an A.B. test for the web would be that you're going to make two different versions of a web page, and then, or more than that, and then try to decide which one is quote-unquote better, and we'll discuss what that means. So, why would you want to A.B. test? There are a lot of reasons. So, if you're trying to get people to sign up for something, and you've got web forms, you might have two different versions of your web form, and hopefully, you're able to optimize that and get more conversions. You get more leads, more sign-ups, more people interested in your product or your service, and hopefully, you're actually getting people more engaged in your site, and then, maybe, they'll want to come back and stick around. So, this might mean more money, but if you're not selling something, it isn't necessarily. This could be good even if you're higher education, a government site, you want people to find your stuff, you want them to be using the site more effectively. So, one of the kind of side benefits is you can also get people on the site longer, and that can actually help with search engine optimization, where your bounce rate is lower, and therefore, you get higher search engine rankings. Hopefully, you get happier users as a result. So, those are some reasons why you would A.B. test. So, what are the types of things that you can A.B. test? There's lots of things. I'm going to focus mostly on, basically, on web testing, but you can do parts of a web page. So, if you wanted to just do, you know, classic is like, I'm going to change the button, right? Everybody's, okay, A.B. test, I'm going to change the button. But, you don't have to just change a button. You could have a completely different page. You can have a page with a completely different layout. It's totally different. There's nothing that looks the same. And actually, that can be a very powerful test. You can also test more than just a single page. You can actually test a whole funnel, they call them, of pages, where the whole funnel is different. You could even have a different number of pages in the funnel. So, one funnel might have three pages, and the other one has two, and you're trying to see where people are dropping off the funnel, if you're losing people, or if they're actually going through the whole process. Another place where people A.B. test is in advertisements. So, Facebook ads, AdWords, that kind of thing. And you have usually less to work with there. Pricing models is a pretty standard thing. I'm sure most of us have gone to the pages where there's a software as a service, and there's like the typical either three or four options there. There's like the cheapy one, and the real expensive in the middle one. And they've probably done A.B. test on the size of the one that they're hoping that you're going to choose, and the color, and that kind of thing. So, there's a lot of work done with the look of the pricing, but also even the numbers. So, they might change the numbers. You might get a different. I think some people might have seen this sometimes when you're going to airlines, and you're like, wait, just two seconds ago, I swear, on my phone, that was not that same price. So, that could be interesting. Another thing is emails. So, if you're doing any kind of email marketing, something with MailChimp, or things like that where you can target things and try to see who's coming to your site, who's clicking on those links based on the type of email. So, if you're looking specifically at a web page, there's lots of things you can change on the web page, right? So, here's an example where I can change hello. I can change the image. I could make it completely different, but if there's just sort of targeted things I want to try out, I could change, you know, text, you know, the layout, add more links, remove them, so forth, and do any kind of combination that I want. So, you can get pretty flexible. So, part of this is talking about how do we do this. And there's many ways you can approach A.B. testing. A pretty standard approach is to use some sort of software as a service. And that's mainly what we're going to talk about today, is using Optimize.ly. But you can do A.B. testing completely with custom code. You don't have to use a service at all. This is Drupal, and if you're a coder, I've actually done this kind of thing in the past, and we have one site where we're doing a combination of software as a service and some custom code, and doing a hybrid approach. So, you know, if you see all the services out there, and you're like, ah, they're too expensive, or they don't do what I want, then, you know, you certainly can roll your own. But there's other aspects of it. You can do a redirect A.B. test, and that's where you have two completely different pages, at least conceptually different pages, where they're different in terms of the URL. And then you split your audience into, say, it was 50-50, 50% go to the original URL, and then 50% go to the redirected page. So that's a classic redirect test. You can use your software as a service, so something like Optimize.ly, and you can use their editor, and we'll be looking at that, where you can go and you can just change all sorts of stuff. So that tends to be popular, especially with people who are non-technical. But believe me, you can do some pretty funky things in there, and you can change HTML, you can change CSS, you could add JavaScript. So in some ways, you're like, oh, the marketing group, you know, or the non-technical group can go in and play with this tool and do some stuff. But, you know, you might end up having some interesting A.B. tests, if that happens. Or you'll be called in to help them with the technical side. You could go in and add, you know, your own JavaScript to make your changes, or do any combination of these things. So obviously you need to pick some tools, and there's a ton, ton, ton out there. We'll talk about Optimize.ly today, but there's many competitors. Aquialift is an interesting tool that does not just A.B. testing, but all sorts of personalization. And actually a lot of the A.B. testing tools are trying to expand more into the whole personalization space as well, which is just providing customized, personalized content to their users no matter what. You just go somewhere and it's tailored to you. Everyone's gone on the web, and you're on some random website looking at Hawaiian vacations, and then you're like an advertisement for Pantheon or for Aquia, and you're like, I'm looking at vacations. Why are they talking? You know, it's because I happen to go and look at Drupal stuff all the time. And so they think for some reason that, you know, that's the kind of ad they should show me. So that's the kind of personalization. You can, I have it personally, but there are A.B. testing frameworks you can use at a code level. So something I found was PHP A.B. It's on GitHub. You could try that out, or you can just write your own code. So an important part of A.B. testing is making sure you understand what is your goal. So what are you trying to test for? So you can make it A.B. test, but then not have a goal, and you're like, oh, well, I don't know. I guess it's better. I mean, that doesn't help, right? So typically a pretty standard thing is they filled in the form and clicked the button, or they just clicked a button, or sometimes it's just how long they've been on the page. That can be some indicator that the page is more useful. But you need to have some sort of goal that you're set up. I would argue that you shouldn't have too many goals. Try to be very selective and be very focused on what goals you want to focus on. All right. So for optimizly, this is one of the software as the service tools that you can use to do A.B. testing. And I'm going to go through the UI. I'm going to do a demo. Hopefully the Internet will let me. So obviously it's a third-party tool. You need to add some JavaScript on your site. Probably how many here are developers? Hands up? Oh, almost everybody, or probably everybody. You know how to add JavaScript to your site. I'm not going to, you know, put it however you want to do it. There's many different ways in Drupal, right? So put it somewhere, and that's how it's going to actually be able to pull in the stuff optimizly. Once you've done that, then you'll be able to, and a note on that is you could be selective and decide only to add it on certain pages of your site if you want to do that, because optimizly does have a little bit of a performance hit, so you could, you know, just opt in for certain pages. So then at that point you're going to create an experiment, which is an A.B. test. And, you know, I'm going to show an example of just one page, and you create, you can create variants. And in my case, I'll just create one. You make whatever changes you want. You save it. There's at least an optimizly. You have to make sure to remember to save, but there's also a publish. So you're like, eh, where's my change? And like, oh, I forgot to hit the publish button. So that's kind of a side note. There's a lot of settings you can do in optimizly, and we'll touch upon some of them, not all of them, because there's a lot there. But you can have many variations. So it's, you don't have to have just A, B. You could have as many as you want if you want to do that. Would only do that if you have enough traffic to be able to support that kind of thing, because you need enough traffic in order to look at the results and decide if something was statistically significant. And then you could do multiple pages if you wanted. We'll look at just one page. One thing that you'll often want to do is look at your audience. So a very standard segmentation is to have desktop versus mobile. So an A, B test, you probably wouldn't want to do them both for all different devices. That's a pretty standard thing, is you'd have an A, B test and you'd actually split it between the two. Then you can do things, you could look at clicks. That's mainly what we'll do, but you can do custom metrics if you want. You can add JavaScript and CSS. And then you can do, the traffic allocation is, how many people do you want to, do you want to just do a split 50-50 between A and B? Or do you want to say, well, this new page is super risky. This is a high traffic page. I'm a little nervous about rolling this out. I'm going to do 90-10, because I just don't know. I might be completely shooting myself on the foot by doing this test. I don't want to lose all my leads overnight by going 50-50, so I'm just going to kind of ease into it. So it's not uncommon for sometimes people to do more risky things, but do it at a lower percentage. And then you can actually schedule A, B tests to roll out at certain times if you want. So once you have configured everything, you start your experiment, you have to wait. And how long you wait depends on how much traffic you have. If you get a lot of traffic, you don't have to wait as long, because you'll have lots of data. If you don't have that much traffic, then you'll have to wait longer. If you have too little traffic, it doesn't make sense to do the test. So you have to judge how many people are hitting the page, and does it even make sense for me to try to do this? So you wait a certain amount of time. The tool will tell you, have you gotten enough data that it makes sense to look at, you say, was it a good test or not? If you're doing A, B, and B wins, then you implement B, and you move on, you do another test. So that tends to be the process. You can have as many A, B tests running at the same time if you want to, you know, all your landing pages, your home page, whatever, all your forms, you could go crazy if you want. So at this point, I'm going to try to demo. We'll see how that works. Maybe not demo the Slack. We'll quit out of that. All right. So, oh, it's so small. Sorry. It looks super tiny up here. All right. So, first thing I'm going to do is show you kind of what my example is. So here is a web page, and it's got some text, it's got an image, it's got some links, and I'd like to change it. I'd like to change it. So it looks like this, because why not? I'm going to change my image, I'm going to change my H1. I'm going to actually, it's hard to tell, but my find out more is actually a slightly darker orange, and I've made orange up in the top on the contact. So I've made actually four different changes on this page. And there's going to, I'm going to show that test, and if I have time, I'll show another one where the original looks like this. And I was like, ah, it's kind of boring. I don't know. I want to play around with it. My new one actually looks completely different. And actually, this one I set up with a redirect test. So the original and the new one are actually at completely different URLs. So I was, I've used Optimize.ly a lot for a certain client and was used to their interface. It's not a great interface, but last week I'm like, well, I can't use the client's account because that's just, you know, shouldn't do that. So I set up a new account. Well, they completely, the new interface is completely different than what I've used. So I've been clicking around and learning the new interface. So hopefully people, I might want to make it a little bit bigger. There we go. So the interface here is, this is Optimize.ly, I guess, 10 for the web or X. Along the left, they have kind of like the bigger buckets of things. And I'll go over those briefly and then we'll dive into experiments themselves. So here I have five experiments I have tested or set up. Like I said, I have one that's like desktop versus mobile. And then let's look a little bit about some of the other pages. So they have a settings. I won't go into too much detail, but you can add jQuery to your config if you want in order to add that when you're adding JavaScript. There's, you know, other things that you can add. You can play with your caching. Like when you're testing, you might want to turn off caching just so it's easier for you to see stuff and then enable that later. So I'm not going to go too much into the settings. As far as other stuff, so audiences, they built in, they have all sorts of audiences you can build from. So if I were to create a new audience, please refresh your browser. Okay. I'm scared. Oh, no. Okay, there's no internet. All right. Well, that's going to be a problem. Let's try that again. Drupalcon. Everyone else needs to log out of the internet. Okay. Well, it's going to be a hard demo when I have no internet. Okay. There you go. Turn it off and on again. Let's try it. Cross your fingers. Yay. Maybe that worked. All right. Give the man a prize. All right. So we've got here, we've got audiences that you could add. I added some earlier, you know, some mobile desktop Austria because why not? And that's kind of a fun thing. It's kind of an interesting interface. You could go in and I want to add, you know, let's see, language, drag and drop. Language is Arabic. And then you can and or them. So very views like or actually a little bit context. So that's kind of fun. You can add your own audiences and use those later. So another thing that they've added is something called attributes where you can actually add your own custom things from your code with JavaScript. So you would say, oh, that like the last presentation was like, oh, have you been at Drupalcon before? So your user has been and it adds some JavaScript and then all of a sudden that properties available to you in your interface, which is kind of cool. They have some extension stuff, which I'm actually not going to go into because we probably won't have time. But let's go back into the experiments because that's kind of the meat and potatoes. So I have this homepage experiment split between desktop and mobile. Take a look at desktop and hopefully the internet will hold up. Okay, good. So you probably saw it flicker, right? It showed off showed the original and then it wrote over it and it's showing the new thing. So we have, I changed four things on this. I changed this contact. I changed the text here. I changed the color on this button and I changed the image. So it's pretty simple. If I wanted to change this text here, all of a sudden we have here, you know, and we can add some stuff, right? And we save it. But there is a publish button as well, which I haven't clicked. So it's very, very simple to change things. You have all sorts of options. So for the image, it just lets you upload a new image. Pretty simple. I swapped it out there. It has all the CSS selectors and it's very straightforward. So to give you some ideas of the things that you can change, you can change layout. You can actually move things around. Actually, for the mobile test, I did that. I moved the image above the text and you could do, you know, position it however you want and relative to a selector, you can add borders. You can change the size of something. You could add your own CSS. In this case, I actually bumped up the image because there was too much white space on the top. I haven't played with the timing. But the track clicks is that's where if you want to actually add another thing, like a goal or event to track. So I actually have that for here. Find out more. We see that for this one, I changed, I think it was the color. I think the background color, I changed the color, made it darker. And then I have a track click there because I want to know if they click that. So for my success here, my goal is that they either click this button or they click this content button. I'm hoping that I'm going to engage them more and they're going to do one of those two things. So those are my goals for that. So it's pretty powerful. You can do all sorts of more things. You can insert HTML, you know, well, that's an HTML, but you know, you have the idea. And then you can say where you want to insert it. So before, I want to insert that before. And then all of a sudden I have this gobbledygook there, right? And then, you know, I could save that if I wanted to. I haven't published. So it's pretty amazing how simple it is. You can insert stuff. You can add CSS. You can insert a new image, insert HTML, change things, which is what I had done here. I had changed different elements. You can do a redirect test, which is another example I have. So it's pretty simple. Some of the other things. So those are variations. You can create as many variations as you want. And so it has the different tabs. So once I'm in a specific experiment, I have another set of tabs on the left. And I've said, okay, I'm using this particular page. I actually tacked on an argument just to differentiate it for myself, for the test. But you can specify the URL there or other pages that you want to affect. And then you can specify what audience is. In this case, I specified it was just for the desktop. And because I have a desktop versus mobile test. Integrations, I did not add anything. You could actually integrate this with Google Analytics or other things if you want, maybe AdWords. But it has a whole list of possible integrations. So Adobe, some of these I'm not familiar with, but some of your standard ones. So have fun with that. And I mean, if you added it with Google Analytics, if that's what you're using, then you can tie it in with your data there and get much better idea of conversions. So the metrics are the things, the goals that you're trying to track. So I set up, click the contact button, click to find out more. And then, well, if you visit the page, that's not such a great goal. But your time on the page would be better. So you have a primary and secondary metrics. And you can get the data for those. You have some shared code that's for all of the different variations, which is kind of interesting. I don't know exactly why you would want to do that, but maybe I added an alert, because why not? And here's where you do the traffic allocation. For testing purposes, I recommend that you actually do a split, zero, 100, if you're doing AB test, because when you're testing, it's very easy to see your variation in real life if you're doing 100%. Don't forget to change it. Once you're actually running the real thing, you'll want to do your 50-50, or your 90-10, or whatever you want to do. But for me, personally, I like to do the 100% while I'm developing to see that make sure it's actually working. Scheduling, if you wanted to do that. And that's kind of all the main things there. So let's take a quick look, got a couple minutes. Take a quick look at mobile, because that's slightly different. The page is the same, but the audience is mobile. So that's the differentiator. So optimally, it's going to split those people up for you. And in the variation, the main difference here, actually, it was, I think, pretty much the same test, except for I bumped the image on top. So if I look at the image, actually layout, I moved it. I just moved it there. So you can have fun and do all sorts of interesting things. Will it necessarily work on every browser? I don't know, probably not. But play with it and try it out and do your normal kind of testing that you would do to make sure that the devices that you care about are supported. So that's like the standard sort of AB test. I'll quickly go over a redirect test. So I had this other one where I was redirecting from this Autodesk by itself to Autodesk-Vienna. And we can quickly take a look at that. So we have a redirect test. And actually have it split up into two, which is kind of interesting. We have the redirect test itself. And it's pretty simple. All it has is a redirect. So it just says, okay, I'm going to send some traffic to this other URL. So in my case, I'm directing them from without the Vienna to the Vienna. And that's it. You can preserve parameters if you want. So that test is super simple, but it has all the same things that you can change as far as the traffic allocation and all that stuff. Where it gets a little more interesting is I said, okay, why don't I layer that on top of having a test that actually changes the thing that I redirected to. So I actually made another test on top of it, which only looks at the one that's Vienna. That's this URL here. And so it has just the Vienna. So I layered them, because why not? And then in there, I just played around and hid stuff and just had fun. So I just got rid of the old title completely and actually added, actually, this is a completely different page. So in Drupal, it's actually physically a different page. So that's why this content is actually completely different here. It's a different page in Drupal itself. But you can also add content within Optimize.ly, too, if you want. So that's fast and furious. I think I'm pretty much out of time. If anyone had a quick question, then welcome to use the mic. Anyone? Okay, cool. I like when you do the A-B test, maybe on a home page, and then do a redirect, the user will recognize them being redirected in the URL. Yeah. So yeah. So be careful, you know, if you want a more seamless experience, sometimes it can be a little bit jarring. And I don't know if you noticed, you can, so if I go to this one, I go there originally, let's see if it does it. See the flicker? So you need to be careful when you're tuning it, especially images, you know, because it's going to load the thing a little bit slower. It is actually loading Drupal twice, right? It loads the disadvantage of using a third party service is it's loading the page and then deciding, oh, you want to change the page? Okay, fine. Let's change the page, right? And it's changing it, especially for the redirect, right? Then it's loading a whole other page. For the non-redirect test, it's just doing some JavaScript on the page, so it's just a one-load thing. But you do need to take that in consideration. You don't want to like, oh, well, now the page is slower to load because they had two renders, and then now our statistics are a little bit goofy. So make sure that you're testing it, that you're not, you know, slowing things down too much that you're affecting the test itself. Any other questions? All right, so I'm just going to... There are some... Here, I'll go back into here. I do have a bunch of resources that are in the slides themselves, so I will put these up on the session page today. There are some other kind of tips that you might want to look at, and I just want to call out to my son. He did all the doodles in the talk, and he actually made this, so we're making stickers for this for Bad Camp in California, so I just want to say thanks to Aaron. And come Friday to the sprints, if you can, and help contribute. You do not have to be a developer, I think, pretty much everyone was, but you don't have to be a developer to help at the sprints, and that's it.