6.2 How does A/B testing work? - Video Tutorials & Practice Problems

<v ->A very important process to use,</v> when you're working with marketing analytics, is what's called an A/B test. So here's a very early example of an A/B test. I was given these slides by one of the early CTOs at Amazon. You've probably never seen the shopping cart on the left side of the screen. But ask yourself, how does Amazon know what should go on the right side of the screen? How did it get there? So now every e-commerce site puts it on the right side of the screen. Well, that was from an A/B test. So the left is A, the right is B. And what you wanna do is test them against each other, show some of the visitors to the site A, show other visitors of the site B and see which one works better. So, how would you decide this at your company? Most companies don't do A/B test nearly as often as they should, and I wonder if yours does. But, how would you decide if you had been at Amazon and needed to decide whether the shopping cart was on the left or the right? So, you could have had everybody get together and vote, or you could've said, hey, it's a sales decision, whatever they wanna do, or it's a design decision, whatever the designer picks, or you might have trouble agreeing and you just compromise and run it down the middle of the page. Well, none of those things are really better than asking your customers. Because all of those are about your opinion, what you really care about is your customer's opinion. And the A/B test allows you to check your customer's opinion. Well, Amazon ran this test back in 1998, and they found out they had a 1% higher conversion rate for running the cart on the right side of the page. And so this has been worth 1% of all of Amazon's revenue since 1998. Because you might say to yourself, well, I know Amazon does A/B testing, but that's because they have lots of money, and we can't afford to do A/B testing. Well, it's not true. This was done in 1998. Amazon didn't have a lot of money back then. But what's really true is the opposite. It's not that Amazon can do A/B testing because they have a lot of money, it's that Amazon does A/B testing and therefore they have a lot of money. So that's what you should be focused on too. How do you make these important decisions? Use A/B testing. You might ask yourself, well, how do I know how long to run an A/B test before it is statistically significant? Well, there are formulas that you can use, there are calculations that will help you do that. But a simple way that you could figure it out, is to do something called an A/A test. Well, what's an A/A test? Well an A/A test says. How about if you run an A/B test, but you use the exact same treatment for both tests? And you might say, well, that's the dumbest test I've ever heard of. If they're the same, what are you learning? What you are actually learning how long it takes to run a test before the numbers for both sides are the same? So, how long does it take to shake the noise out of the system? Running an A/A test can give you that answer, at least on a estimated basis. So an A/B test can power your improvement of the customer experience on your site. Take a look at this example. This is an example that was given to me for one of my books. And basically what it shows, is two very different approaches. And one of them, had 116% higher conversion than the other. And maybe looking at it, I don't know if you would know which one it is, but it's actually the one that was on the right. But it took the months to figure this out. There were all sorts of variations that they did. And so you might look at this and say, oh, I could tell the one on the right is better. Well, they didn't know the one on the right was better. They went through dozens and dozens of different variations and tests, before they figured that out. And if they're experts in their website and it took them that long, it will probably take you that long to solve your problem as well. Don't underestimate how long it can take for A/B testing to really find the right answer for you. Be patient, keep coming up with variations, keep testing it. And eventually, maybe your conversion rate will be 116% higher as well. What people do instead of A/B testing is they focus on best practices. So they try and figure out, hey, what's the best thing to do? Some book told me to do this. Some expert told me to do this. We hired a consultant and they said to do this. That's actually not really good enough. You really shouldn't be focused just on what best practices are, because optimization is not about best practices. The only thing best practices can be good for, is it can tell you what to start to test, but best practices are hardly ever best for you. They might be best practices in general, but for your customers, for your website, for your situation, an A/B test is really the only way that you're going to determine what's the best practice for you. And when you're A/B testing, don't only test single pages. Those are the easiest test to run, changing the color of a button or a picture or some marketing copy, but you can also test full journeys. You can rearrange pages, you can reduce the number of pages, you can change the way they're connected to each other. Testing those full journeys can give you a lot of important improvement in your customer conversion rate, that you won't get if you only test individual pages. When you're A/B testing, what you're really trying to do, is to find the middle ground. So what do we mean by the middle ground? Well, it's the middle ground between what customers want and what you want. That's actually the middle ground, that's gonna make your website really work. And so finding that middle ground is what helps you continuously improve your marketing. Every A/B test gets you closer to that middle ground. So let's look at an example of an A/B test. And I wanted to pick an unusual example, something you might not have thought about A/B testing. What about A/B testing, the ranking algorithm in your site search engine? So, what does this test tell you? So suppose you've got two different dashboards, one for each ranking algorithm, and suppose you picked a 90/10 split. So 90% of your searches, are going to your production search engine with your existing ranking algorithm, that you know is working to a relatively good effectiveness. But we wanted to test the change. We wanna try changing the ranking algorithm. Maybe what we wanna do is to say, that the title of the page is a way more important place to find that search keyword that was entered than the rest of the page. And we wanna really boost, pages that have the keyword found in the title. And let's say that that's what we do with that ranking algorithm. And we siphoned off 10% of the searches to a different search engine that's got that new ranking algorithm. Well, which one's working better? Well, suppose we have a dashboard here that can show you when people are finding what they're looking for. And it can show you, that for the first search engine, people are finding what they're looking for at a 40% success rate. But maybe for the second search engine, your new search engine with this new brilliant algorithm you came up with. Well, now we're finding that it's two points worse. So the test algorithm isn't better than production, in fact, it's hurting the success rate. So what kinds of questions should you ask? Well, the first question to ask, is how long should I run it before I actually trust that the second algorithm is worse? And one of the other questions you might wanna ask is. What happens when you see something turn negative or what happens when something is positive? Well, maybe what you might wanna do to answer these questions, is to use a calculator, to tell you when you've reached statistical significance. If your dashboard is showing that the new algorithm is two points worse than the old algorithm, you might not even want to wait until you reach statistical significance. Because, what you're looking for is for the new algorithm to be better. You don't actually have to prove to yourself that it's worse. It could just be that you're not sure. And so you might wanna move on to another test. The other thing you can do. Is if you're seeing that the algorithm is close, but you still haven't reached statistical significance and you really wanna know which one's better than the other. You could try moving the split. So instead of sticking with a 90/10 split, maybe move to 75/25 or even 50/50. That will help you to get to that answer much quicker, because the more you can split the test, the more repetitions you'll get with both algorithm and you will then get the answer faster. So this is a whole set of techniques for you to use around A/B testing, which is one of the most important processes you can use in improving your marketing, using your analytics.