Fundraising: This Is Not a Test
Package testing in direct mail is fundamental and essential. It's how you determine which packages provide your best return on investment. Even more important, it's how you make sure every element of your program — lists, offer, copy and design — is continually evolving to produce the best results, even as times and circumstances change. Everybody agrees on that.
Yet, even with so much at stake, organizations and individuals too often go for the short money by trying to dumb down their testing into anecdotal comparisons. Nowadays the "T word" has been bandied about so much, actual tests are getting lost in the shuffle.
Accurate, statistically valid testing is a rigorous process. It needs to be conducted by people who not only know how to do it, but who also know how to properly analyze the results. The methodology for conducting a reliable and ongoing testing program can go way beyond a simple A/B split, and as a humble copywriter, I dare not get into the weeds of it here. There are plenty of experts who can help you plan a long-term testing strategy.
But after some years of working with those folks, and of observing the nonprofit world in general, I've had plenty of opportunity to seen what testing is not. For example:
- If you mailed two different acquisition packages at the same time, then abandoned the idea that had the lower response rate … this is not a test.
- If you mailed an annual fund appeal in June 2013 and measured the results against the certificate package you mailed in 2012 … this is not a test.
- If someone says, "We mailed that package 10 years ago, and it didn't do well," … this is not a test.
- If you mailed your acquisition control to half your file, then, as a test, tried a different teaser because you're tired of the old one, included a shorter letter because "people don't read anymore," and at the last minute, threw in a window cling because the control has been fatiguing … this is not a test.
- If you try to minimize your risk by keeping all your best donors in the regular mailstream, instead of letting the test panel accurately reflect your entire file … this is not a test.
- The following are also not tests:
- Educated guesses
- Gut feelings
- Comparing response rates and average gift amounts, without considering critical factors like net income, cost to acquire a donor or cost to raise a dollar
Every comparison is not a test. But because accurate testing can be tedious and sometimes expensive, many people don't do it and say they did.
To save some cash in the short run, they fall back on the shortcuts above and hope the results will be accurate enough to get by. And they might be. But it isn't testing, so they'll never really know. Worse, they won't have the information they need to confidently move forward, knowing that one success is leading to another, and another …
Willis Turner believes great writing has the power to change minds, save lives, and make people want to dance and sing. Willis is the creative director at Huntsinger & Jeffer. He worked as a lead writer and creative director in the traditional advertising world for more than 15 years before making the switch to fundraising 20 years ago. In his work with nonprofit organizations and associations, he has written thousands of appeals, renewals and acquisition communications for every medium. He creates direct-response campaigns, and collateral communications materials that get attention, tell powerful stories and persuade people to take action or make a donation.