To anyone in Direct mail, testing is nothing new, and it may even be something that you do all the time. But are you testing correctly? Many mailers make common mistakes that make their testing less than optimal or even cause them to base their decisions on incorrect data.

The key to proper testing is to think empirically. Let’s take a ride on the ‘wayback machine’ all the way to high school science class and scientific method. But what does science have to do with direct mail you say? Well, that’s a fantastic question – thanks for putting your hand up and asking!

Scientific method states that you start with a hypothesis and go about proving or disproving it in a controlled and repeatable fashion. OK, don’t start staring out the window just yet – stay with me a little longer.

So in our case our hypothesis might be that a colored envelope is better than a standard white envelope. For our test we would use the white envelope as our control and the colored envelope as our test. Since we only want to know if envelope color matters, we need to limit our testing to just that one variable. That means you keep every other element of the offer the same and try to make both mailing segments and all other variable as close to identical as you can.

Now this is where someone usually says “well, why don’t we also try different teaser text on the colored envelope?” How about NO!

It’s so easy to be led astray by trying a few elements at once but all you’re doing is spoiling the test by adding in another variable. Don’t do it. Don’t even think about it. If you take a lax attitude to testing you may end up being led astray by the lies of bad data.

The scientific method to testing may seem like it will take a long time to learn anything, but if used effectively after just 3 tests you will know the following:

  • My colored envelope works better than the white one
  • The version A envelope teaser text works better than the version B text
  • The version B offer out performs the version A offer

After just 3 tests you have an optimized package that can be used as a control to test any number of new ideas. So go ahead and be creative, but also use logic. There’s a reason why Spock was the science officer.

So the next time someone asks you what you do, go ahead and tell them you’re basically a scientist. Not the kind that cures Polio but the kind that mails awesome direct mail packages.