Monitoring and evaluation...we're in it together

It’s a daunting task to begin measuring the effectiveness of your work, both in relation to knowing what to measure and what you might discover. The key to success is the involvement of funders and service users. Exploring the factors that lead to poor performance in current monitoring and evaluation practice offers some supporting evidence.

In recent years some funders (they will remain unnamed) have designed monitoring and evaluation systems in isolation from the projects and services they fund, and then handed them to service deliverers to complete. As a result, these systems feel more like an imposition, rather than a vital check and balance of how our projects and services are performing. At best we feel demotivated, it feels like a chore, and at worst we feel frustration at being asked to measure the wrong things.

Far from entering into a discussion to create a system that works for funders, service deliverers, and service users alike, we tend to jump through hoops by half heartedly completing the monitoring and evaluation reports we inherit. Of course, this keeps the funders happy, which subsequently keeps funding rolling in. But take a longer term view of the situation. It gives funders the false impression that they are collecting the right information, and it hampers our ability to conduct effective monitoring and evaluation. We fail to learn and apply lessons because we lack the relevant data and motivation to do so. Ultimately, it’s our service users that are affected.

However, we are often guilty of committing the same error when it comes to our relationship with stakeholders. If we develop indicators and outcomes in consultation with our funders, we can be a little too convinced our own wisdom. There’s a temptation to second guess what is important to our service users, rather than just asking them. We’re experts, right? This approach can lead to vital information being overlooked, and it can have a knock-on effect on the services we provide. WRVS’s project to capture what is important illustrates this point perfectly.

Until two years ago WRVS measured the effectiveness of its work largely by measuring numbers, for example, how many meals on wheels were delivered on target. The senior management team took the bold decision to ask the question ‘so what?’ They wanted to know what difference their services actually made to service users’ lives. After commissioning researchers to ask service users what really matters to them, WRVS found that some participants in the meals on wheels programme did not eat the meals they received, yet they continued to participate, because the human contact left them feeling less isolated. WRVS still measure the number of meals delivered, but their monitoring and evaluation is now geared towards measuring softer outcomes, such as reduced isolation and increased confidence.

There are, of course, numerous pressures on both funders and service deliverers, which make for an imperfect relationship when it comes to monitoring and evaluation. However, both parties ultimately strive to achieve the same goal: a better standard of living for the people they support. Rather than entrenching old notions of what counts as success, we need to recognise that effective monitoring and evaluation is dependent on the involvement of both funders and service users.

New Philanthropy Capital will release a new publication this month which will explore what funders can do to help charities conduct monitoring and evaluation and demonstrate impact more effectively. Helping grantees focus on impact will be available to download for free from New Philanthropy Capital's website from 16th March

Clothes collections: needing a makeover?

The fundraising technique of clothes collections may be a traditional mainstay for our sector, but has been recently having some perception issues. To donors, charity clothes collections represent a convenient way of giving away items that are no longer wanted; yet the increase in bogus collections means that members of the public can sometimes find it hard to tell a legitimate ask from an unscrupulous one.

To charity fundraisers the collection of clothes for reuse or resale can be a crucial income generator - yet it’s becoming an increasingly crowded marketplace, with donors often receiving many requests from charities to donate items in this way. Add into the mix recent sensational and negative media reports on charity clothes collections, and the public perception of this fundraising form becomes somewhat blurred.

It is heartening to see these issues being taken up at the highest level, with a Government roundtable debate on bogus collections meaning that a joined up discussion with all relevant stakeholders present and engaged took place at the end of last year. The Institute will soon be bringing its House-to-House Code of Fundraising Practice which clarifies the standards required for the charity sector and donors alike.

What is crucial in all of this is for the sector to work together in presenting a united front. We need to acknowledge that there are a range of models of clothes collections which it might suit a particular charity to employ and that there is no standard rule as to which arrangement is better. It depends on the individual circumstances and make up of the charity in question.

At the end of the day, what is at stake is the giving public’s levels of trust and confidence in charities. If there is uncertainty around techniques, what constitutes a legitimate fundraising collection, how charities use donations of clothes and the profits they keep from such gifts, people’s comprehension of the benefits of this form of giving will naturally decline.

Of course, individuals are more inclined to make judgments about a charity based on what they see of its activity first hand. Some people may have experienced the work of a good cause through being a service user or beneficiary. However, more often than not it’s fundraising that is the mode of engagement with individuals. And it stands to reason that the most ‘visible’ forms of fundraising, such as house-to-house, or face-to-face are shop windows for the rest of the sector’s income generating activity.

This all feeds neatly into the Institute’s accountability and transparency agenda; openness around all issues where fundraising is concerned – costs, salaries and required investment alike - is always encouraged. There are some basics of fundraising which should be covered off as standard by charities. One such example is the premise that money raised needs to be used by charities as intended by donors.

There are no two ways about it. The tense times we are still experiencing as a sector mean an increase in charities’ demand for funding. Crucially, it’s up to all of us to fulfill the missions of our organisations and, in all of the fundraising methods we use, engender trust in service users and donors alike.

Louise Richards, Director of Policy and Campaigns, Institute of Fundraising

Does your impact have impact?

As a matter of transparency, all charities should host their annual report on their websites, but let’s be honest, not many people trawl through these reports, so it might be better to start elsewhere. If you want to begin improving accountability and transparency, start by thinking about who your stakeholders are, what they want to know, and what the best way to communicate with them is.

Each group of stakeholders has a slightly different perspective on your organisation, they will therefore want different information. Unfortunately, there isn’t a one-size-fits-all approach to demonstrating impact; you need to present information in a variety of ways if it is to hit its mark. Funders will want detailed analysis of the outputs and outcomes of your projects, beneficiaries may look for evidence that they are being listened to, and that improvements to existing services have been made. Similarly, supporters and staff may want ‘softer’ information, for example, anecdotal evidence of the change your organisation has made to people’s lives. But don’t guess what they want, ask them!

A good place to begin is to consider the kind of language you use to explain the difference your organisation makes. The language used in monitoring and evaluation reports is fine for funders, they have to read them, but they’re paid to do so. If you use the same language in impact literature, then you are likely to send your audience to sleep, or worse still, alienate them. It’s encouraging for stakeholders to hear that your organisation has hit all its targets, but it’s not very inspiring. This is the point at which you need to take off your performance hat and put on your marketing hat. The aim is to breathe some life into what can often be quite dry information. Think about using snappy quotes to illustrate qualitative information, there is nothing quite as powerful as a beneficiary explaining the improvement in the quality of his or her life.

Next is the amount of time a person will spend reading about the impact your organisation has made. At the risk of appearing to slide into an infinite regress, you have to be aware of how much of an impact your impact is making. What do I mean? Ask yourself how many impact reports you have received over the last year, and then how many you have read, and then how many you remember. I imagine for most of us it’s a dismally small number.

Room to Read, an NGO that builds schools and libraries in developing countries, has a novel approach to tackling this problem. It too produces an annual impact report, but it also demonstrates its impact by including a very brief ‘live’ summary of achievements on it website and below the signature of every email it sends out. At time of writing it read ‘Our impact: 1,128 schools, 10,000 libraries, 433 books published, 7.4 million books distributed, 8,944 girls scholarships, 4.1 million children benefited’. This information is included one the 'About Us' section of their website in every email a Room to Read member of staff sends, and with offices all over the world this means that their impact message is delivered thousands of times each day. Granted, it lacks the emotional edge of anecdotal evidence, but it is nonetheless a very powerful and effective way to demonstrate impact.

Demonstrating the difference your organisation makes creates a lasting positive impression, but it’s not easy. Finding new and inventive ways to get your message across can be challenging, but as Room to Read shows, it’s not impossible, and it need not be expensive.

The third sector needs a dose of cautious optimism

What do you feel when you hear the words accountability and transparency? I fear that to some ‘being accountable’ is the feeling of sheer panic that sets in when you realise there is only one week remaining to submit your return to the Charity Commission (I’m sure very few of you do this, of course). And ‘being more transparent’ is something that is demanded of an organisation that has been up to no good. Yet to many the whole process is about setting out your stall and shouting ‘look what we’ve done!’

It’s the latter attitude that underpins the work of the ImpACT Coalition, a positive ethos of openness and honesty. The Coalition takes the view that third sector leaders should seek to embed accountability and transparency practices into the culture of their organisations as a matter of routine, and that we should dispense with the narrow view of accountability as a process that happens once a year.

Clearly, this is a lot easier said than done. Being accountable implies responsibility to another party, and, like any responsibility, it is often accompanied by the fear of failure. It therefore follows that many well intentioned leaders are reluctant to improve accountability and transparency because of a fear of failure.

But what are we really afraid of? I think it breaks down into two broad concerns: one internal to the organisation and the other external. Firstly, to make an organisation more transparent one must review internal processes, and this could mean, among other issues, uncovering poor performance. How would you explain this to your colleagues and to the board? How would you tell your longest serving employee that his project has been underperforming for the past 10 years? It doesn’t take long for the objections to stack up.

Second is the fear of exposure, or more precisely, the fear of being judged too harshly. When you open up your organisation to greater scrutiny what will people think and say? What if others think what you consider to be a success a failure? What if people criticise your strategic plan? Worse still, what if you actually have bad news to deliver? Such concerns accumulate to create a feeling of a loss of control over your organisation’s reputation, and who wants that?

These concerns are well founded. I would suggest that any responsible leader would - indeed should - have similar concerns, but simply being concerned about the right issues is not enough. The most adept leaders reframe concerns as challenges and overcome them by adopting an attitude of cautious optimism.

So how would a cautious optimist go about improving accountability and transparency? Well, when it comes to the possibility of uncovering poor performance, she would first communicate to the staff and board that it’s about a collective effort to improve communication with stakeholders, she would emphasise that it’s about asking what we can learn, and would not rush to ascribe blame (of course, if the poor performance continues once identified, then action needs to be taken!).

Building an atmosphere of openness is absolutely critical to the success of improving accountability and transparency: external transparency is a reflection of internal transparency.

And what about exposure? It must be understood that the choice on offer are not limited to zero transparency one day and absolute transparency the next; we’re not out to expose poor practice for the sake of it (this should be reserved for persistent offenders!). The cautious optimist would commit to improving accountability over a period of time to mitigate potential fallout, with the ideal goal being the highest level of transparency in the shortest time possible. It would be unfair to expect an organisation with, say, limited resources and a modest turnover to expose every aspect of its operations at the drop of a hat without first allowing it to make improvements.

We do, of course, actively encourage organisations to share what they have learned from their failures, but such lessons are learned in the months spent reflecting after a failure, not in the midst of it. Demanding absolute transparency at such an early stage for the sake of it makes people defensive, exacerbates fears, and ultimately makes organisations less likely to be transparent.

This isn’t to suggest that we shouldn’t aim to be completely transparent, we should, and we should hold ourselves accountable to the highest standard. But if we want others to take up the challenge, we must be sensitive to the fears that accompany increased transparency, and understand that the first steps are often the most daunting; they are certainly the most critical.

The ImpACT Coalition wants third sector leaders to make a genuine commitment to improve accountability and transparency and not just a token gesture (a ‘commitment’ to improve over a five year period holds no weight). Improving accountability and transparency should be driven by a genuine desire to communicate more effectively with stakeholders, not just because it looks good.

It is this cautious yet optimistic approach to accountability and transparency in the early stages that ultimately dictates its success, and the ImpACT Coalition exists to support organisations on this journey. Is it daunting? Yes. Is it challenging? Yes. It is worth it? Absolutely!