Whether you’ve been doing something the same way for years or are working on launching something new, the only way to know for certain if one approach is better than another is to run a test. In fact, any time you’re using a marketing tactic that provides measurable results, you should consider testing. The following case study demonstrates the real-world value of this.
Background
For the past ten years MacKenzie has been sending an in-mail survey to customers of a major player in the powersports industry. Surveys are sent to everyone who has made a purchase in the past two years – currently a list of over 30,000 people. Recipients are given the option of either completing the paper survey and mailing it back, or going online to take the survey there.
Problem
Each direct mail package has always included a $1.00 bill, included to “thank recipients in advance” for completing the survey. With over 30,000 surveys being mailed out each year, the $1.00 bill incentive clearly gets very expensive! Naturally, our client wanted to know if they could switch to a less expensive incentive and still keep response rates level.
Solution
MacKenzie designed an A/B/C split test. One-third of the list was sent the $1.00 bill; one-third received an iPad sweepstakes give-away offer in which 5 survey respondents would be randomly chosen to win a free iPad; and one-third received both the $1.00 bill and the sweepstakes offer.
Results
As it turns out, moving to the “sweepstakes only” offer decreased response by over 40%, while including both offers produced an 8% increase in completed surveys:
|
Offer |
Response Rate |
| $1.00 Bill |
12% |
| iPad Sweepstakes |
7% |
| $1.00 Bill + iPad Sweepstakes |
13% |
So, although we had all hoped to get a different result, our A/B/C split test ended up validating what we had been doing all along. In this case, the test justified this very expensive line item in the client’s budget. As the Market Research Association says has been shown in many other surveys, getting something tangible right away simply has an intrinsic value that a possible future incentive can’t match – even when you’re comparing dollar bills to $400 iPads.
