A Nonprofit Horror Story

In a recent blog, we talked about the negative consequences of evaluating nonprofits on the basis of their “overhead ratios,” the ratio of the organization’s operating costs to its income. Although useful when looked at in conjunction with other aspects of the organization’s data, when taken alone, it can be downright misleading. Worse, the overhead ratio can “wag the dog,” becoming an overriding goal of the nonprofit, rather than the success of its mission.

Social psychologist Donald Campbell described how when any single quantitative indicator is used for decision-making or evaluation, it is more likely to distort the processes it is intended to inform. Known as “Campbell’s Law,” this principle has been widely cited in education to criticize the sole reliance on student test scores to evaluate the quality of schools. Test scores wind up wagging the dog, music and art budgets get cut, and teachers sometimes cheat. In the same way, using overhead ratios as a single measure of the quality of a nonprofit can distort the picture dramatically.

So how might one better evaluate the quality of a nonprofit? Borrowing another term often found in education, one might look at its efficacy, the organization’s capacity to produce a desired outcome. In short, its effectiveness. Nonprofits rarely look beyond the quantity of resources (monies) that are provided through its programs or services to examine a more fundamental question… Do the things we are funding actually make a difference? Do they work?

That’s why it caught our attention last year when a major international development charity did something unusual—it publicly announced that one of its programs didn’t work.

The U.S.-based nonprofit Evidence Action scales anti-poverty programs around the world. One of the programs it funds is No Lean Season, which assists poor Bangladeshis in the off-season between planting and harvesting. During these three months, families have no income, and many go hungry. Worldwide, this problem is widespread, as many as 300 million rural poor may be the victims of seasonal poverty.

The No Lean Season program gave small subsidies to farmers, so they could migrate to cities between September and November to find jobs for the months before the harvest. Pilot programs had gone well. For as little as a $20, subsidy people went to cities and found jobs, sent their earnings home and returned for the harvest. They migrated the following year even in the absence of a subsidy.

Based on these early results, the program was scaled up. In 2016, the program was introduced in 82 villages. One year later, it was offered to 699. No Lean Season was listed on Givewell as a top charity.

Randomized Controlled Trial

Then Evidence Action wanted more data to assess the program’s efficacy. No Lean Season was subjected to a rigorous randomized controlled trial (RCT) to evaluate the benefits of giving the subsidies at such a large scale. RCTs are considered to be the gold standard for this kind of program analysis.

The results of the study were both surprising and disappointing. Evidence Action announced in a blog post, “An RCT at scale found that the (No Lean Season) program did not have the desired impact on inducing migration and, consequently, did not increase income or consumption.”

It’s rare for a nonprofit to participate in this kind of research at all, much less find that its program doesn’t work and announce it to the public. Although the study demonstrated that there was a small increase in migration, the overall efficacy of the program was poor. Evidence Action used the data to make adjustments to the program and is currently reevaluating to see if these changes altered the outcomes. In the meantime, Evidence Action contacted Givewell and informed it that No Lean Season should not be listed as a top charity program and suspended fundraising for it.

Pretty scary stuff, eh? Halloween-level, horror movie scary. For any nonprofit, this kind of admission has the potential to spook donors and wind up hurting the very cause that the organization is seeking to help. And many nonprofits don’t have multiple programs in operation like Evidence Action does. Announcing anything less than shout-it-from-the mountaintop success could be an existential threat to the organization.

Despite the fear factor, we believe that rigorous efficacy research is invaluable. What’s important is not to look at research outcomes as a thumbs up or thumbs down verdict, but rather as information the organization can use to improve on the delivery of its mission. Delivery on the mission needs to be considered an iterative process, fine-tuned and reassessed regularly.

It’s true that most nonprofits don’t have the budget to fund efficacy research programs. But there are ample opportunities for partnerships. For example, nonprofits with health-care missions are naturals to attract research collaboration with university schools of public health. Graduate students beat the bushes for research topics; coming to a school of public health with this kind of opportunity would be like arriving home with presents on Christmas morning.

A nonprofit’s overhead ratio is irrelevant if the dollars that are provided for its mission doesn’t produce meaningful change. At best overhead ratios are a piece of the overall picture; at worst they’re a smokescreen. We believe that more information about programs’ efficacy will strengthen rather than diminish donor affinity. The entire nonprofit sector would benefit if every organization was as diligent and transparent as Evidence Action.