Fundraising: This Is Not a Test
Package testing in direct mail is fundamental and essential. It's how you determine which packages provide your best return on investment. Even more important, it's how you make sure every element of your program — lists, offer, copy and design — is continually evolving to produce the best results, even as times and circumstances change. Everybody agrees on that.
Yet, even with so much at stake, organizations and individuals too often go for the short money by trying to dumb down their testing into anecdotal comparisons. Nowadays the "T word" has been bandied about so much, actual tests are getting lost in the shuffle.
Accurate, statistically valid testing is a rigorous process. It needs to be conducted by people who not only know how to do it, but who also know how to properly analyze the results. The methodology for conducting a reliable and ongoing testing program can go way beyond a simple A/B split, and as a humble copywriter, I dare not get into the weeds of it here. There are plenty of experts who can help you plan a long-term testing strategy.
But after some years of working with those folks, and of observing the nonprofit world in general, I've had plenty of opportunity to seen what testing is not. For example:
- If you mailed two different acquisition packages at the same time, then abandoned the idea that had the lower response rate … this is not a test.
- If you mailed an annual fund appeal in June 2013 and measured the results against the certificate package you mailed in 2012 … this is not a test.
- If someone says, "We mailed that package 10 years ago, and it didn't do well," … this is not a test.
- If you mailed your acquisition control to half your file, then, as a test, tried a different teaser because you're tired of the old one, included a shorter letter because "people don't read anymore," and at the last minute, threw in a window cling because the control has been fatiguing … this is not a test.
- If you try to minimize your risk by keeping all your best donors in the regular mailstream, instead of letting the test panel accurately reflect your entire file … this is not a test.
- The following are also not tests:
- Educated guesses
- Gut feelings
- Comparing response rates and average gift amounts, without considering critical factors like net income, cost to acquire a donor or cost to raise a dollar
Willis believes in expressive writing, exceptional fundraising, and exuberant living.
Willis Turner is the senior copywriter at Huntsinger & Jeffer. He was an experienced writer and creative director in the traditional advertising world for more than 20 years before making the switch to fundraising nearly 15 years ago. In his work with nonprofit organizations and associations, he has written thousands of appeals, renewals and acquisition communications for every medium. He creates direct-response campaigns, as well as collateral materials and communications, that get attention, tell emotional stories, and persuade people to take action or make a donation.