Written by Pete

Much has been written on this subject… OK, I’m lying. Almost no-one has written anything on this. Which makes it all the more bizarre, when you consider just how powerful these arrays are as a tool for multivariate tests. If you just want the xls files so you can get on and play, you can get them here. If you also want to know how they work, what they’re doing and how to make the most of them, read on.

PAM-VAR Testing

The Pareto principle states that for any system, 20% of the outcomes generated will come from 80% of the population of variables. Or to put it another way, 80% of what you get comes from 20% of what you do.

This allows us to logically conclude that a representative sample of any specific population will allow us to estimate the results for the rest of it. In medicine, this is performing a biopsy, in mechanical engineering it’s called Taguchi testing, and in web based evaluation, I’ve termed it PAM-VAR testing (Pareto Analysis of Multi-Variate Array Results).

The key to this method is determining what constitutes a representative sample. For example, if we wanted a sample of numbers from 1 to 100, you wouldn’t pick 1, 2, 3, 4, 5, 6, 7, 8, 9 and 10. Instead, you’d go for middle of range values, so you’d go for a set more like 3, 15, 24, 37, 45, 56, 67, 73, 86 and 95.

But how do we do that, when we’ve got a PPC campaign to test, with 5 variations of headline, body copy and URL? The simple answer is that we borrow from our friends in engineering.

Taguchi Arrays

Turns out some clever bugger called Genichi Taguchi had come across this problem some time before me, and invented something called the Taguchi orthogonal array. Taguchi’s brilliant line of thinking ran something like this:

“If we can make the results of a set of tests mimic the most extreme variations that we’d expect to get, we can estimate the top and bottom percentage of results by sampling a small number of the total possible potential variations.” Basically, Taguchi figured that if taking a small lump of someoneís liver could tell you about the rest of it, you could take a small sample of potential figures for variables in an equation, and use them to estimate the outcomes of all the others.

What Arrays Do We Use?

Personally, I like symmetry, so the arrays I use work on matched numbers of variables. That means that if I’m testing three headlines, I’ll also test five variates of body copy and three URLs. It’s perfectly possible to test a set of two three and three, or three four and five, or any other set you can come up with. However, the arrays are slightly harder to construct. Nevertheless, if you want to generate spreadsheets or applications to calculate these, it’s certainly doable, and you could reverse engineer it from the xls files available here, if combined with the arrays from FreeQuality.org.

Application in PPC Multivariate Tests

So, imagine we’ve got a keyword group in a PPC campaign we’ve set up. We’ve got five different versions of the title, copy and URL, and we want to test to find the optimum version for that set of keywords. If we wanted to test every single combination, that would mean running 125 different adverts. Now imagine if you wanted to do this across 5 sets of keyword groups. OR even worse, 3 groups of 5 sets of keywords. All of a sudden, you’re having to create and test 625 ads in the first instance, and 1,875 in the second. That’s simply not practical.

Fortunately, by applying the tools we’ve provided, you can trim this to a fifth of the normal figures. Obviously this is hugely beneficial, as it makes these kinds of large-scale tests practical, as you’re running 125 and 375 permutations, instead of the figures shown above. Whilst these are obviously still large numbers, they’re far smaller and more manageable.

Application in Other Online Multivariate Tests

Now, imagine if you were to run this same test across a web design, or a long copy sales letter, where you’ve got four, five or more variables. All of a sudden, you can be looking at huge values. For instance, a multivariate web design test with five variables, and four permutations of each would take just 16 tests using our system. That’s against a normal size of 1,024. Or if you wanted to test 5 variables with 5 options each, that would give 25 samples for testing, instead of the normal 3,125.

Again, you could work out how to construct the test array from the xls files available here, combined with the Taguchi arrays from FreeQuality. If however you’re completely lazy, we’ll be coming back to this next week, to show you how to do that. And we’ll probably build an online version of this in the future to make it even easier.

Talk To Us

If you’ve found this useful, please leave feedback in the comments below.