By Jon Huggett

The third sector is awash in measures that do not work.  A passion for “accuracy” has produced metrics that are at best complicated to produce and hard to understand.  Many rely on so many assumptions that their accuracy is easily challenged.  And worse, they get in the way of innovation.

Paradoxically, measures that drive innovation are often simple, even though they can often be questions by any ‘expert’.

We all want good measures.  Governments around the world are showing interest in the third sector, as long as it can ‘prove that it delivers results’.  Donors ask NGOs to show their results.  The people that work inside NGOs want to know that they make a difference.  A chief executive of one of the world’s top NGOs recently shared with me that their internal measures are lacking.

There is an industry producing measures for the third sector.  It churns out an alphabet soup of SROI and so on.  Experts meet at conferences to discuss how to value impact.  Analysts strive for accuracy. 

The problem is that few of these measures are actually much use. 

If we keep asking the same question, but do not get a reasonable answer, are we asking the right question?

One question to start with is ‘which measures work’? 

Which measures drive innovation?

Which measures can decision makers use day-to-day?

Which measures bring in money?

What metrics could we use to assess these assorted measures?

Consider innovation.  We all believe in it.  But how do we know if something is simply a fad that will fade with the next fashion, or a step forward?

A desire to ‘prove’ that a small entrepreneurship is working can sometimes drives us to ‘pull the plant up by its roots to see what is going on’.  Yet for start-ups, a few simple questions can be very revealing.  Six questions that SIX asks of new ventures are:

1  What's new?

2  Why is it good?

3  Who does it help now?

4  How does it work?

5  How much does it cost?

6  Where can it go now?

Large NGOs can be innovative, too.  Habitat for Humanity International measures its impact by ‘families served’.  It changed from ‘homes built’ as it is easier to house people by renovating existing building in places such as Bombay where land is scarce.  The change in the measure prompted the US organisation to examine why it was spending $100,000 per new home in a land with an oversupply of housing.  The measure of ‘families served’ is not perfect, and some may argue that it is quite flawed.  But it has helped Habitat for Humanity International develop new, better, cheaper ways to house more people.

“Proof” that something works may help raise money and reassure donors that their money is well spent.  But it may not help make day-to-day decisions.  The desire for a “proof” makes it hard to try new ideas.

Detailed, robust longitudinal studies have shown how effective the programs of some third-sector organisations can be.  Sesame Workshop, Big Brothers Big Sisters and Nurse Family Partnership have each shown that their programs work and produce benefits that can be tracked over many years.  And the research has helped them raise significant money.  However, this research does not help that much in deciding whether a change is an improvement or not, nor for that matter in working out where to put resources day-to-day..

At the heart of the problem is that successful innovation is usually the result of lots of trial and error.  Simple measures that give quick feedback allow people to learn swiftly.  Short cycle times are at the heart of most innovative organizations.  This is true as well in technology and science.  John Chambers, the CEO of CISCO in its early days, always wanted to put a prototype in front of a customer early: “quick and dirty, cheap and easy … then make it better.”  And scientists have been known to say that genius is 10% inspiration and 90% perspiration.

In the third sector much innovation comes from people at the front line.  An acid test of whether a measure helps innovation is the conversation that it stimulates between two field leaders the first time they meet.  If the conversation moves from the measure to ‘how did you do that’, it’s helping transmit ideas.  If the conversation moves to a discussion of context – ‘its different in our country’ or a questioning of the measure itself, it’s not helping. 

Some measures that aid innovation are maddeningly simple, or even simplistic.  Publication of school test results has shown how different schools with different approaches can have different results.  Much of the educational orthodoxy is being challenged, sometimes found wanting, and sometimes found to be robust.  As any teacher will tell you, tests are crude, and are not the only way to know how education is contributing to individuals or society at large.  But when schools, or children, get very different test results, it is reasonable to ask ‘why’.

In the same way, the speedometer on a car does not measure the quality of a car’s engine.  But if one car can accelerate much faster than another that looks the same, and maintain a much higher top speed, too, it would be reasonable for a technician to wonder what is going on under the bonnet.

Some measures of consumer behaviour, too, can be highly effective, if simple.  “Net promoter scores” are a rapid way to find out whether people who recommend a service or product to friends.  They simply ask people to give a rating on a scale of 1-10, even in a text message.  They give quick comparisons, prompt questions like “why” and “how”, which may lead to more investigations.

So let’s start to assess whether our measures are helping innovation, or whether a desire for accuracy, a fear of being wrong, is getting in the way of innovation.