A Nonprofit Horror Story
In a recent blog, we talked about the negative consequences of evaluating nonprofits on the basis of their “overhead ratios,” the ratio of the organization’s operating costs to its income. Although useful when looked at in conjunction with other aspects of the organization’s data, when taken alone, it can be downright misleading. Worse, the overhead ratio can “wag the dog,” becoming an overriding goal of the nonprofit, rather than the success of its mission.
Social psychologist Donald Campbell described how when any single quantitative indicator is used for decision-making or evaluation, it is more likely to distort the processes it is intended to inform. Known as “Campbell’s Law,” this principle has been widely cited in education to criticize the sole reliance on student test scores to evaluate the quality of schools. Test scores wind up wagging the dog, music and art budgets get cut, and teachers sometimes cheat. In the same way, using overhead ratios as a single measure of the quality of a nonprofit can distort the picture dramatically.
So how might one better evaluate the quality of a nonprofit? Borrowing another term often found in education, one might look at its efficacy, the organization’s capacity to produce a desired outcome. In short, its effectiveness. Nonprofits rarely look beyond the quantity of resources (monies) that are provided through its programs or services to examine a more fundamental question… Do the things we are funding actually make a difference? Do they work?
That’s why it caught our attention last year when a major international development charity did something unusual—it publicly announced that one of its programs didn’t work.
The U.S.-based nonprofit Evidence Action scales anti-poverty programs around the world. One of the programs it funds is No Lean Season, which assists poor Bangladeshis in the off-season between planting and harvesting. During these three months, families have no income, and many go hungry. Worldwide, this problem is widespread, as many as 300 million rural poor may be the victims of seasonal poverty.
The No Lean Season program gave small subsidies to farmers, so they could migrate to cities between September and November to find jobs for the months before the harvest. Pilot programs had gone well. For as little as a $20, subsidy people went to cities and found jobs, sent their earnings home and returned for the harvest. They migrated the following year even in the absence of a subsidy.
Based on these early results, the program was scaled up. In 2016, the program was introduced in 82 villages. One year later, it was offered to 699. No Lean Season was listed on Givewell as a top charity.
Randomized Controlled Trial
Then Evidence Action wanted more data to assess the program’s efficacy. No Lean Season was subjected to a rigorous randomized controlled trial (RCT) to evaluate the benefits of giving the subsidies at such a large scale. RCTs are considered to be the gold standard for this kind of program analysis.
The results of the study were both surprising and disappointing. Evidence Action announced in a blog post, “An RCT at scale found that the (No Lean Season) program did not have the desired impact on inducing migration and, consequently, did not increase income or consumption.”
It’s rare for a nonprofit to participate in this kind of research at all, much less find that its program doesn’t work and announce it to the public. Although the study demonstrated that there was a small increase in migration, the overall efficacy of the program was poor. Evidence Action used the data to make adjustments to the program and is currently reevaluating to see if these changes altered the outcomes. In the meantime, Evidence Action contacted Givewell and informed it that No Lean Season should not be listed as a top charity program and suspended fundraising for it.
Pretty scary stuff, eh? Halloween-level, horror movie scary. For any nonprofit, this kind of admission has the potential to spook donors and wind up hurting the very cause that the organization is seeking to help. And many nonprofits don’t have multiple programs in operation like Evidence Action does. Announcing anything less than shout-it-from-the mountaintop success could be an existential threat to the organization.
Despite the fear factor, we believe that rigorous efficacy research is invaluable. What’s important is not to look at research outcomes as a thumbs up or thumbs down verdict, but rather as information the organization can use to improve on the delivery of its mission. Delivery on the mission needs to be considered an iterative process, fine-tuned and reassessed regularly.
It’s true that most nonprofits don’t have the budget to fund efficacy research programs. But there are ample opportunities for partnerships. For example, nonprofits with health-care missions are naturals to attract research collaboration with university schools of public health. Graduate students beat the bushes for research topics; coming to a school of public health with this kind of opportunity would be like arriving home with presents on Christmas morning.
A nonprofit’s overhead ratio is irrelevant if the dollars that are provided for its mission doesn’t produce meaningful change. At best overhead ratios are a piece of the overall picture; at worst they’re a smokescreen. We believe that more information about programs’ efficacy will strengthen rather than diminish donor affinity. The entire nonprofit sector would benefit if every organization was as diligent and transparent as Evidence Action.
Otis Fulton, Ph.D., spent most of his career in the education industry, working at the psychometric research and development firm MetaMetrics Inc., Pearson Education and others. Since 2013, he has focused on the nonprofit sector, applying psychology to fundraising and donor behavior at Turnkey. He is the co-author of the 2017 book, ”Dollar Dash: The Behavioral Economics of Peer-to-Peer Fundraising,” and the 2023 book, "Social Fundraising: Mining the New Peer-to-Peer Landscape," and is a frequent speaker at national nonprofit conferences. With Katrina VanHuss, he co-authors a blog at NonProfit PRO, “Peeling the Onion,” on the intersection of psychology and philanthropy.
Otis is a much sought-after copywriter for nonprofit fundraising messages. He has written campaigns for UNICEF, St. Jude’s Children’s Research Hospital, March of Dimes, Susan G. Komen, the USO and dozens of other organizations. He has a Ph.D. in social psychology from Virginia Commonwealth University and a Bachelor of Arts from the University of Virginia, where he also played on UVA’s first ACC champion basketball team.
Katrina VanHuss has helped national nonprofits raise funds and friends since 1989 when she founded Turnkey. Her client’s successes and her dedication to research have made her a sought-after speaker, presenting at national conferences for Blackbaud, Peer to Peer Professional Forum, Nonprofit PRO, The Need Help Foundation and her clients’ national meetings. The firm’s work is underpinned by the study and application of behavioral economics and social psychology. Turnkey provides project engagements, coaching, counsel and staffing to nonprofits seeking to improve revenue or create new revenue. Her work extends into organizational alignment efforts and executive coaching.
Katrina regularly shares her wit and business experiences on her and Otis Fulton's NonProfit PRO blog “Peeling the Onion.” She and Otis are also co-authors of the books, "Dollar Dash: The Behavioral Economics of Peer-to-Peer Fundraising" and "Social Fundraising: Mining the New Peer-to-Peer Landscape." When not writing or researching, Katrina likes to make things — furniture from reclaimed wood, new gardens, food with no recipe. Katrina’s favorite Saturday is spent cleaning out the garage, mowing the grass, making something new, all while listening to loud music by now-deceased black women, throwing in a few sets on the weight bench off and on, then collapsing on the couch with her husband Otis to gang-watch new Netflix series whilst drinking sauvignon blanc.
Katrina grew up on a Virginia beef cattle and tobacco farm with her three brothers. She is accordingly skilled in hand to hand combat and witty repartee — skills gained at the expense of her brothers. Katrina’s claim to fame is having made it to the “American Gladiator” Richmond competition as a finalist in her late 20s, progressing in the competition until a strangely large blonde woman knocked her off a pedestal with an oversized pain-inducing Q-tip. Katrina’s mantra for life is “Be nice. Do good. Embrace embarrassment.” Clearly she’s got No. 3 down.