Massive Multivariate Testing: The PAM-VAR Tool

Posted by Pete - 19/11/08 at 08:11:49 pm

Earlier this month, I presented at SMX London on a tool we developed for optimising PPC campaigns. However, that tool was created about six months ago, and would only do one thing - optimise PPC copy.

Well, since then we’ve sat down, had a think, and come up with a new one. However, it’s a little more difficult to use we thought we’d take a moment to fill you in on what it does, why it’s cool, and how to use it.

What It Does

The new tool consists of two parts; an array generator for building your test, and then a form builder which creates the correct form for entering your results and calculating the results.

The principles behind what’s being done can be seen in these two posts.

Why It’s Cool

What this tool lets you do is run massive multivariate tests, and generate usable data, without having to run every possible combination. For instance, if we look at a test where we’ve got four variables (the number of things being tested), and five treatments (the number of versions of each thing you’re testing), you’d normally need to run 625 different tests. However, using this tool, you’d only need to run 16.

The upshot of this is that you can test almost anything you can think of. Want to test how PPC copy works with landing pages and payment systems all in one go? Now you can. Want to test five different elements of a landing page, when you have five variations of each bit? That’s possible too.!

In fact, using this tool, you can test up to five variables with five treatments each! That’d normally require 3,125 tests, but we can reduce that to just 25!

How to Use It

Firstly, head over to the tool. As you can see, you first need to enter the number of variables and treatments. Then click the Generate Array button, and you’ll get the array.

Once you’ve been given that, you need to know what to do with it. So if you head over there now, you’ll see the interface give you two dropdowns, one for selecting the number of variables, and one for selecting the number of treatments. Once you’ve picked these, you’ll be given a grid. Here’s where it gets a little bit complicated.

Firstly, each column corresponds to a particular variable, so column one is variable one, column two is variable two and so on. The number in the cell is the treatment to use, and finally each row is one test.

Thus row one, cell one dictates which treatment of variable one to use. Row one, cell two shows the treatment to use for variable two, and so on.

Using the grid you’re given, set up your tests. Then run them, collect your data and come back.

You then go through the process again, and click the Enter Results button. At this point you’ll be presented with a list of form fields to enter your data into. Enter the conversion figures for each test into the fields, and click Run Calculations, and hey presto, you’ll be shown which treatments worked best for each variable.

A Word of Warning

Whilst this is very useful, and does give usable data, it’s not perfect. The only way to be certain that one treatment worked best would be to run every possible combination. What this does is say which one is probably best, to a reasonable degree of accuracy. Therefore, once you’ve got the data for what the tool says is the best combination, go and test that in some A/B tests against results you already have. That will validate it properly.

That said, I’ve personally had great results with this, and I hope you do too!

Get a Trackback link

1 Trackbacks/Pingbacks

1. Pingback: The Big List of PPC Resources & Articles | The Marketr on December 21, 2008

Search