Fundraising: This Is Not a Test

Package testing in direct mail is fundamental and essential. It's how you determine which packages provide your best return on investment. Even more important, it's how you make sure every element of your program — lists, offer, copy and design — is continually evolving to produce the best results, even as times and circumstances change. Everybody agrees on that.

Yet, even with so much at stake, organizations and individuals too often go for the short money by trying to dumb down their testing into anecdotal comparisons. Nowadays the "T word" has been bandied about so much, actual tests are getting lost in the shuffle.

Accurate, statistically valid testing is a rigorous process. It needs to be conducted by people who not only know how to do it, but who also know how to properly analyze the results. The methodology for conducting a reliable and ongoing testing program can go way beyond a simple A/B split, and as a humble copywriter, I dare not get into the weeds of it here. There are plenty of experts who can help you plan a long-term testing strategy.

But after some years of working with those folks, and of observing the nonprofit world in general, I've had plenty of opportunity to seen what testing is not. For example:

1. If you mailed two different acquisition packages at the same time, then abandoned the idea that had the lower response rate … this is not a test.
2. If you mailed an annual fund appeal in June 2013 and measured the results against the certificate package you mailed in 2012 … this is not a test.
3. If someone says, "We mailed that package 10 years ago, and it didn't do well," … this is not a test.
4. If you mailed your acquisition control to half your file, then, as a test, tried a different teaser because you're tired of the old one, included a shorter letter because "people don't read anymore," and at the last minute, threw in a window cling because the control has been fatiguing … this is not a test.
5. If you try to minimize your risk by keeping all your best donors in the regular mailstream, instead of letting the test panel accurately reflect your entire file … this is not a test.
6. The following are also not tests:

• Educated guesses
• Gut feelings
• Comparing response rates and average gift amounts, without considering critical factors like net income, cost to acquire a donor or cost to raise a dollar

Every comparison is not a test. But because accurate testing can be tedious and sometimes expensive, many people don't do it and say they did.

To save some cash in the short run, they fall back on the shortcuts above and hope the results will be accurate enough to get by. And they might be. But it isn't testing, so they'll never really know. Worse, they won't have the information they need to confidently move forward, knowing that one success is leading to another, and another …