Is Your Nonprofit Addicted to Vanity Metrics?

Lean Impact

You might have heard the buzz term ‘vanity metrics’ before. And even if you haven’t, you can guess what they’re about – metrics that give you warm fuzzies, but don’t have an impact on your mission. In the nonprofit sector, it can be hard measuring impact – but with a thinning pipeline of funds, showing direct outcomes for donated dollars is more important than ever. For that reason, an increasing number of organizations (knowingly or not) look to vanity metrics to justify the funding they currently receive and make the case for more.

Not everything about ‘vanity metrics’ is relevant to nonprofits, so I’ll break down the applicable pieces below. I’m also going to pose some critical questions for us to consider as nonprofits work hard to make a difference, while competing to fund that work.

What Are Vanity Metrics?

Vanity metrics are those sneaky numbers that make you feel good about the work you’ve done, but don’t get you closer to achieving your mission. Because they show the illusion of progress, they prevent us from truly measuring impact and seeing what’s working and what isn’t. Especially when we fail to see the latter, we miss the chance to “pivot” (change our strategy) or throw in the towel on a method that simply doesn’t work.

At my old nonprofit job, I remember writing grant reports where we counted things like number of school clubs, attendance at events, number of events attended, etc. Our numbers always looked great, and they steadily increased year by year. One time though, I was driving home with a student from an exhausting 4-day event. I asked him what kind of campaigns he was working on at his school and he replied, much to my surprise, “I’m too busy with [your organization’s] activities to do a campaign at my school right now.”

I was shocked, but I shouldn’t have been. Of course our programs were making important change, but it only took one conversation to see that only focusing on vanity metrics had kept us from realizing that a key part of our program model was broken. In this case, the vanity metrics were the raw number of attendees and raw number of trainings. However, we didn’t collect the right data to see how one cohort of students were using the ideas they learned from the trainings, which was the core part of our mission.

How Else Do Vanity Metrics Show Up In Nonprofits?

Have you ever gotten excited over 10 people liking your Facebook page (but then those same people never engage with your organization in any other way)? Be honest. A little bit, right? If so, you’ve fallen victim to a vanity metric. Here are a few examples of places where vanity metrics show up in nonprofits:

Grant reporting – Foundations ask us for metrics, so we give them the ones that will make us look good – how many attendees at a conference? How many hours of technical assistance? Unfortunately, often enough these reports are full of vanity metrics and do not show us whether (or not) we are fulfilling our mission.
Social media – We can count Facebook likes, retweets, followers, and shares all we want, but are those numbers turning into action towards our missions? How do you know?
Annual reports – Nonprofits use annual reports to explain their success to funders and foundations. What numbers go into those reports? Some funders ask for challenges you’ve faced, but there’s always a pressure to paint a picture far brighter than the reality of a program or product that isn’t doing what you thought it would.

Measuring Impact Right – Actionable Metrics

In The Lean Startup, Ries’ describes ‘actionable metrics’, which he claims are the antidote to vanity metrics. Actionable metrics have the 3 A’s: actionable, accessible, and auditable.

Actionable metrics demonstrate clear cause and effect on your mission. If x is growing, then we are achieving y towards our mission.
Accessible metrics are easy to understand. Remember, metrics are people, too. We can’t just take a group of numbers – 10 more events per year, 40 more attendees per event, etc., and call that a “success.” When measuring impact we must ask, what’s the story behind those numbers? Storytelling brings numbers to life. And cohort-based reporting can turn complex actions into people-based reports. For example, follow a participant through an entire program cycle – what are the outcomes? And don’t forget to make your data accessible to every member of your organization.
Auditable metrics use good data. Is your data credible? Does it have tons of mistakes? If you have a huge data set, make sure to double-check your conclusions with real people. Don’t come to a conclusion based on data without asking a real person whether it matches their experience.

Actionable metrics tell the true story of the impact of our organization. And if our actionable metrics aren’t showing progress, then it’s time to think about whether to continue (persevere) or change your strategy (pivot).

Are Nonprofits Addicted To Vanity Metrics?

At a systems level, the current nonprofit structure does not support actionable metrics well at all. Simply put, the funding structure of the nonprofit system gives nonprofits more incentives to celebrate vanity metrics than to use actionable metrics. Jennifer Ceema Samimi, in her essay “Funding America’s Nonprofits,” explains this structure:

“[The Nonprofit Sector is] an industry that requires organizations to compete for government and foundation funding. Known as the Nonprofit Industrial Complex (NPIC), this system forces nonprofits to professionalize, wherein they must focus on maintaining funding rather than fulfilling their mission.”

An organization must do all it can to prove its programs are working in order to maintain funding. Given that system, radically changing strategy or taking risks on new initiatives are out of the question. And vanity metrics are shiny trophies that are easier to hold up, rather than asking tougher questions and using actionable metrics to find out the truth.

At the individual level, we are also part of upholding the Nonprofit Industrial Complex (NPIC). Especially for nonprofit staff that doesn’t get to do direct service work, celebrating vanity metrics often become what make them feel like their work is having an impact. Unfortunately, these members of the staff are also usually the ones with the responsibility of illustrating impact and answering to funders. In the end, vanity metrics are a huge barrier to utilizing actionable metrics.

What’s Next? A Way Forward

We must create a culture in which it is okay for nonprofits to fail. If a nonprofit isn’t working, it’s wasting precious resources, and it needs to stop, pivot or fold. In the long run, those small failures free up money for huge wins. This means changing difficult things within our organizations, and everyday, and in the fundamental structure of our sector. It means using actionable metrics to find out what isn’t working, and if things aren’t working, finding the courage to pivot. From a systems perspective, nonprofits must find new funding models for their ideas (including more grassroots fundraising) and foundations must embrace failure as a healthy part of the social change ecosystem.

Your turn: Let’s be honest here – Do you think your organization is addicted to vanity metrics? How is your organization measuring impact to ensure you are achieving your mission? Please share below in the Comments section!

Want to learn more? Visit us here.