Case Study: The Value of Testing

by Jenny on December 11, 2012 Comments Off on Case Study: The Value of Testing

Whether you’ve been doing something the same way for years or are working on launching something new, the only way to know for certain if one approach is better than another is to run a test. In fact, any time you’re using a marketing tactic that provides measurable results, you should consider testing. The following case study demonstrates the real-world value of this.


For the past ten years MacKenzie has been sending an in-mail survey to customers of a major player in the powersports industry. Surveys are sent to everyone who has made a purchase in the past two years – currently a list of over 30,000 people. Recipients are given the option of either completing the paper survey and mailing it back, or going online to take the survey there.


Each direct mail package has always included a $1.00 bill, included to “thank recipients in advance” for completing the survey. With over 30,000 surveys being mailed out each year, the $1.00 bill incentive clearly gets very expensive! Naturally, our client wanted to know if they could switch to a less expensive incentive and still keep response rates level.


MacKenzie designed an A/B/C split test. One-third of the list was sent the $1.00 bill; one-third received an iPad sweepstakes give-away offer in which 5 survey respondents would be randomly chosen to win a free iPad; and one-third received both the $1.00 bill and the sweepstakes offer.


As it turns out, moving to the “sweepstakes only” offer decreased response by over 40%, while including both offers produced an 8% increase in completed surveys:


Response Rate

$1.00 Bill


iPad Sweepstakes


$1.00 Bill + iPad Sweepstakes



So, although we had all hoped to get a different result, our A/B/C split test ended up validating what we had been doing all along. In this case, the test justified this very expensive line item in the client’s budget. As the Market Research Association says has been shown in many other surveys, getting something tangible right away simply has an intrinsic value that a possible future incentive can’t match – even when you’re comparing dollar bills to $400 iPads.


Share this post:


Jenny Dinnen is President of Sales and Marketing at MacKenzie Corporation. Driven to maximize customer's value and exceed expectations, Jenny carries a can-do attitude wherever she goes. She maintains open communication channels with both her clients and her staff to ensure all goals and objectives are being met in an expeditious manner. Jenny is a big-picture thinker who leads MacKenzie in developing strategies for growth while maintaining a focus on the core services that have made the company a success. Basically, when something needs to get done, go see Jenny. Before joining MacKenzie, Jenny worked at HD Supply as a Marketing Manager and Household Auto Finance in their marketing department. Jenny received her undergrad degree in Marketing from the University of Colorado (Boulder) and her MBA from the University of Redlands.

JennyCase Study: The Value of Testing