Monitoring and evaluation...we're in it together

It’s a daunting task to begin measuring the effectiveness of your work, both in relation to knowing what to measure and what you might discover. The key to success is the involvement of funders and service users. Exploring the factors that lead to poor performance in current monitoring and evaluation practice offers some supporting evidence.

In recent years some funders (they will remain unnamed) have designed monitoring and evaluation systems in isolation from the projects and services they fund, and then handed them to service deliverers to complete. As a result, these systems feel more like an imposition, rather than a vital check and balance of how our projects and services are performing. At best we feel demotivated, it feels like a chore, and at worst we feel frustration at being asked to measure the wrong things.

Far from entering into a discussion to create a system that works for funders, service deliverers, and service users alike, we tend to jump through hoops by half heartedly completing the monitoring and evaluation reports we inherit. Of course, this keeps the funders happy, which subsequently keeps funding rolling in. But take a longer term view of the situation. It gives funders the false impression that they are collecting the right information, and it hampers our ability to conduct effective monitoring and evaluation. We fail to learn and apply lessons because we lack the relevant data and motivation to do so. Ultimately, it’s our service users that are affected.

However, we are often guilty of committing the same error when it comes to our relationship with stakeholders. If we develop indicators and outcomes in consultation with our funders, we can be a little too convinced our own wisdom. There’s a temptation to second guess what is important to our service users, rather than just asking them. We’re experts, right? This approach can lead to vital information being overlooked, and it can have a knock-on effect on the services we provide. WRVS’s project to capture what is important illustrates this point perfectly.

Until two years ago WRVS measured the effectiveness of its work largely by measuring numbers, for example, how many meals on wheels were delivered on target. The senior management team took the bold decision to ask the question ‘so what?’ They wanted to know what difference their services actually made to service users’ lives. After commissioning researchers to ask service users what really matters to them, WRVS found that some participants in the meals on wheels programme did not eat the meals they received, yet they continued to participate, because the human contact left them feeling less isolated. WRVS still measure the number of meals delivered, but their monitoring and evaluation is now geared towards measuring softer outcomes, such as reduced isolation and increased confidence.

There are, of course, numerous pressures on both funders and service deliverers, which make for an imperfect relationship when it comes to monitoring and evaluation. However, both parties ultimately strive to achieve the same goal: a better standard of living for the people they support. Rather than entrenching old notions of what counts as success, we need to recognise that effective monitoring and evaluation is dependent on the involvement of both funders and service users.

New Philanthropy Capital will release a new publication this month which will explore what funders can do to help charities conduct monitoring and evaluation and demonstrate impact more effectively. Helping grantees focus on impact will be available to download for free from New Philanthropy Capital's website from 16th March

Does your impact have impact?

As a matter of transparency, all charities should host their annual report on their websites, but let’s be honest, not many people trawl through these reports, so it might be better to start elsewhere. If you want to begin improving accountability and transparency, start by thinking about who your stakeholders are, what they want to know, and what the best way to communicate with them is.

Each group of stakeholders has a slightly different perspective on your organisation, they will therefore want different information. Unfortunately, there isn’t a one-size-fits-all approach to demonstrating impact; you need to present information in a variety of ways if it is to hit its mark. Funders will want detailed analysis of the outputs and outcomes of your projects, beneficiaries may look for evidence that they are being listened to, and that improvements to existing services have been made. Similarly, supporters and staff may want ‘softer’ information, for example, anecdotal evidence of the change your organisation has made to people’s lives. But don’t guess what they want, ask them!

A good place to begin is to consider the kind of language you use to explain the difference your organisation makes. The language used in monitoring and evaluation reports is fine for funders, they have to read them, but they’re paid to do so. If you use the same language in impact literature, then you are likely to send your audience to sleep, or worse still, alienate them. It’s encouraging for stakeholders to hear that your organisation has hit all its targets, but it’s not very inspiring. This is the point at which you need to take off your performance hat and put on your marketing hat. The aim is to breathe some life into what can often be quite dry information. Think about using snappy quotes to illustrate qualitative information, there is nothing quite as powerful as a beneficiary explaining the improvement in the quality of his or her life.

Next is the amount of time a person will spend reading about the impact your organisation has made. At the risk of appearing to slide into an infinite regress, you have to be aware of how much of an impact your impact is making. What do I mean? Ask yourself how many impact reports you have received over the last year, and then how many you have read, and then how many you remember. I imagine for most of us it’s a dismally small number.

Room to Read, an NGO that builds schools and libraries in developing countries, has a novel approach to tackling this problem. It too produces an annual impact report, but it also demonstrates its impact by including a very brief ‘live’ summary of achievements on it website and below the signature of every email it sends out. At time of writing it read ‘Our impact: 1,128 schools, 10,000 libraries, 433 books published, 7.4 million books distributed, 8,944 girls scholarships, 4.1 million children benefited’. This information is included one the 'About Us' section of their website in every email a Room to Read member of staff sends, and with offices all over the world this means that their impact message is delivered thousands of times each day. Granted, it lacks the emotional edge of anecdotal evidence, but it is nonetheless a very powerful and effective way to demonstrate impact.

Demonstrating the difference your organisation makes creates a lasting positive impression, but it’s not easy. Finding new and inventive ways to get your message across can be challenging, but as Room to Read shows, it’s not impossible, and it need not be expensive.