Does your impact have impact?

As a matter of transparency, all charities should host their annual report on their websites, but let’s be honest, not many people trawl through these reports, so it might be better to start elsewhere. If you want to begin improving accountability and transparency, start by thinking about who your stakeholders are, what they want to know, and what the best way to communicate with them is.

Each group of stakeholders has a slightly different perspective on your organisation, they will therefore want different information. Unfortunately, there isn’t a one-size-fits-all approach to demonstrating impact; you need to present information in a variety of ways if it is to hit its mark. Funders will want detailed analysis of the outputs and outcomes of your projects, beneficiaries may look for evidence that they are being listened to, and that improvements to existing services have been made. Similarly, supporters and staff may want ‘softer’ information, for example, anecdotal evidence of the change your organisation has made to people’s lives. But don’t guess what they want, ask them!

A good place to begin is to consider the kind of language you use to explain the difference your organisation makes. The language used in monitoring and evaluation reports is fine for funders, they have to read them, but they’re paid to do so. If you use the same language in impact literature, then you are likely to send your audience to sleep, or worse still, alienate them. It’s encouraging for stakeholders to hear that your organisation has hit all its targets, but it’s not very inspiring. This is the point at which you need to take off your performance hat and put on your marketing hat. The aim is to breathe some life into what can often be quite dry information. Think about using snappy quotes to illustrate qualitative information, there is nothing quite as powerful as a beneficiary explaining the improvement in the quality of his or her life.

Next is the amount of time a person will spend reading about the impact your organisation has made. At the risk of appearing to slide into an infinite regress, you have to be aware of how much of an impact your impact is making. What do I mean? Ask yourself how many impact reports you have received over the last year, and then how many you have read, and then how many you remember. I imagine for most of us it’s a dismally small number.

Room to Read, an NGO that builds schools and libraries in developing countries, has a novel approach to tackling this problem. It too produces an annual impact report, but it also demonstrates its impact by including a very brief ‘live’ summary of achievements on it website and below the signature of every email it sends out. At time of writing it read ‘Our impact: 1,128 schools, 10,000 libraries, 433 books published, 7.4 million books distributed, 8,944 girls scholarships, 4.1 million children benefited’. This information is included one the 'About Us' section of their website in every email a Room to Read member of staff sends, and with offices all over the world this means that their impact message is delivered thousands of times each day. Granted, it lacks the emotional edge of anecdotal evidence, but it is nonetheless a very powerful and effective way to demonstrate impact.

Demonstrating the difference your organisation makes creates a lasting positive impression, but it’s not easy. Finding new and inventive ways to get your message across can be challenging, but as Room to Read shows, it’s not impossible, and it need not be expensive.

Being the change we want to see...

The ImpACT coalition is shortly to launch its transparency and accountability manifesto, calling on leaders across the sector to get serious about the commitment to being honest and open about our business and how we operate.

Number 8 on the list is for organisations to agree that it is important that they publish an easily accessible suggestions and complaints procedure for beneficiaries and supporters.

I was reminded of the necessity for this only too clearly whilst visiting an elderly relative recently. She told me that a volunteer had been helping her from a local elderly care charity. Whilst the volunteer had been very successful at helping her get the benefits she deserved she didn’t like the way the volunteer often talked to benefits staff on the phone, “she was very rude to them and quite aggressive and on more than one occasion she said I was her mother, which I didn’t really like”. When asked whether she had complained her story was sadly too familiar, “I didn’t like too. She did get me the money I need; besides, I didn’t really know how… she gave me lots of leaflets about how to complain about my benefits but nothing about her organisation.”

It strikes me that we are often in the position of trying to advocate on behalf of those we work with, strengthening and empowering them, but we afford them the same view of our organisations are we are hoping they will get of the statutory services.

Morally most charities would find it hard to argue, but maybe we should also go further. What about publishing how many complaints a charity has each year? A short summary of the type of things and how the organisation plans to address them… True accountability starts at home; maybe it is time for us to take the lead in being the change we want to see…

Beware Comparisons

In April I spend a couple of weeks in the US attending various conferences and meeting with ACEVO’s partners. The topic of charity evaluation was mentioned in pretty much every single meeting.

At the GEO conference in Pittsburgh, foundations were fighting it out over the extent to which they should evaluate the performance of their grantees and what methods they should use. (Although interestingly they were rather less vocal about having their own performance evaluated.) While I was there I learned of two new donor advisory websites to add to the plethora on both sides of the Atlantic. (The new additions to the list are Philanthropedia and Givewell.)

There is a lucrative industry developing to service the sector’s evaluation, impact assessment, and performance management needs. This is of course welcome. There is no question that we need to be better at generating the evidence of the difference we make, and knowing as much as we can about what works and why. However, while much of this work is vital, some is harmful and we need to be clear what we are evaluating, why and who is doing it.

Firstly we need to ask why a charity wants to evaluate its own work. There are two main reasons. One is to get better and doing what you do, learning which programmes work and which don’t, and making sure your money goes where it adds the most value. The second is to convince others outside of the organisation (eg donors, funders, partners) of the good that you do.

These two motivations ought to be able to complement each other as long as the evaluation is sufficiently focused on assessing how well the organisation has achieved its mission. The right performance management framework should produce data which you can share with the outside world to show the impact you have had. However, it is unwise to focus only on gathering data to share with the outside world as you may end up with two systems, and if your performance management system is not concentrating on delivering your mission then it is not fit for purpose.

Secondly there is the question of who is doing the evaluating. A good performance management and outcome measurement framework developed for a particular organisation will tell you something about how well that organisation achieves its mission. (A great example of this is the work of St Giles Trust with Pro Bono Economics.)

Alternatively evaluations by be conducted by or through a third party often seeking to be a broker or a guide to help donors or funders identify the most impactful organisations.

In my view there is a huge amount of value in the work like that undertaken by St Giles. The evaluation they were able to develop demonstrated the outcomes they knew were important for the achievement of their mission. There is a danger though in evaluation frameworks being imposed on organisations where they may not fit, and particularly in being used to compare the relative impact of different organisations.

We should focus our attention on improving performance management in organisations, which would allow each organisation to communicate more effectively the difference they make. We should be wary of efforts to compare organisations for some key reasons.

Firstly, what choices are we actually talking about? If you are worried about the homeless in your home town and there is only one organisation supporting them then who are you going to compare them with? The only real choice is between this organisation and a potential organisation which doesn’t exist in that area. It would be wrong to automatically assume that for each need there is a choice of charities working to meet it.

Even if there is a choice of organisations, for example supporting excluded young people in your area, the approaches and philosophies of those organisations, driven by their missions, may be very different. One may see sport as the route to employment, one may see faith as a way of staying out of prison. Both Kids Company and Eastside Young Leaders Academy, for example, specialise in building the self confidence of troubled children and teenagers. However, their philosophies couldn’t be more different. Both produce the same results (well adjusted young people who can rise to their potential) yet if you swapped the performance management frameworks around both would fail by each other’s standards. A potential donor would be deceived in thinking that there is a straight choice between these two organisations.

Secondly we have to dig a little deeper into what actually motivates people to give. I am not a professional fundraiser but very few people who give to charity do so because they have read evaluations and come to a dispassionate objective judgment. People give because they have an emotional relationship with a cause and in many cases also with an organisation.

Paul Carttar who heads the Social Innovation Fund at the White House talked at the GEO Conference about the dilemma of on the one hand wanting to focus money at the innovations which work and yet on the other the reality that people value organisations in other ways. That does not mean that we shouldn’t prove to donors that we use their money wisely (and indeed keep that conversation live and fresh), but it may mean that we have to think more about how we use qualitative data as well as quantitative, in other words get better at telling our stories. This is much more sophisticated than a traffic light or a percentage score on a comparison website.

Even for more formal processes of choosing between organisations such as a public procurement process, I think we have to be realistic that hard evaluative data is still only part of the picture and subjective personal judgements come into play. Chris Stone, Head of the Hauser Institute at Harvard argued when we met that evaluation reports are unlikely to be what really secures the deal. Public officials are going to be thinking things like ‘do I like and trust the CEO?’, or ‘has the organisation delivered before?’

Thirdly, there is the danger of thinking that one methodology for evaluation or comparison is the holy grail. Tools like SROI are very useful, but they have their limits (as shown by NPC) and it is important that we understand those limits so we use them effectively. We have all railed against inappropriate methods being used to judge charities (such as overhead costs) but we are in danger of reinventing others if we don’t use the right tools in the right context.

The truth is that comparing charities is really really difficult. But even if it were possible it may still not be desirable. We are much better employed on getting better at demonstrating the difference that organisations make in working towards their missions. The Impact Coalition will shortly be launching its transparency manifesto and member organisations will commit to greater accountability and transparency in their work. This is the best start to achieve that goal.

The third sector needs a dose of cautious optimism

What do you feel when you hear the words accountability and transparency? I fear that to some ‘being accountable’ is the feeling of sheer panic that sets in when you realise there is only one week remaining to submit your return to the Charity Commission (I’m sure very few of you do this, of course). And ‘being more transparent’ is something that is demanded of an organisation that has been up to no good. Yet to many the whole process is about setting out your stall and shouting ‘look what we’ve done!’

It’s the latter attitude that underpins the work of the ImpACT Coalition, a positive ethos of openness and honesty. The Coalition takes the view that third sector leaders should seek to embed accountability and transparency practices into the culture of their organisations as a matter of routine, and that we should dispense with the narrow view of accountability as a process that happens once a year.

Clearly, this is a lot easier said than done. Being accountable implies responsibility to another party, and, like any responsibility, it is often accompanied by the fear of failure. It therefore follows that many well intentioned leaders are reluctant to improve accountability and transparency because of a fear of failure.

But what are we really afraid of? I think it breaks down into two broad concerns: one internal to the organisation and the other external. Firstly, to make an organisation more transparent one must review internal processes, and this could mean, among other issues, uncovering poor performance. How would you explain this to your colleagues and to the board? How would you tell your longest serving employee that his project has been underperforming for the past 10 years? It doesn’t take long for the objections to stack up.

Second is the fear of exposure, or more precisely, the fear of being judged too harshly. When you open up your organisation to greater scrutiny what will people think and say? What if others think what you consider to be a success a failure? What if people criticise your strategic plan? Worse still, what if you actually have bad news to deliver? Such concerns accumulate to create a feeling of a loss of control over your organisation’s reputation, and who wants that?

These concerns are well founded. I would suggest that any responsible leader would - indeed should - have similar concerns, but simply being concerned about the right issues is not enough. The most adept leaders reframe concerns as challenges and overcome them by adopting an attitude of cautious optimism.

So how would a cautious optimist go about improving accountability and transparency? Well, when it comes to the possibility of uncovering poor performance, she would first communicate to the staff and board that it’s about a collective effort to improve communication with stakeholders, she would emphasise that it’s about asking what we can learn, and would not rush to ascribe blame (of course, if the poor performance continues once identified, then action needs to be taken!).

Building an atmosphere of openness is absolutely critical to the success of improving accountability and transparency: external transparency is a reflection of internal transparency.

And what about exposure? It must be understood that the choice on offer are not limited to zero transparency one day and absolute transparency the next; we’re not out to expose poor practice for the sake of it (this should be reserved for persistent offenders!). The cautious optimist would commit to improving accountability over a period of time to mitigate potential fallout, with the ideal goal being the highest level of transparency in the shortest time possible. It would be unfair to expect an organisation with, say, limited resources and a modest turnover to expose every aspect of its operations at the drop of a hat without first allowing it to make improvements.

We do, of course, actively encourage organisations to share what they have learned from their failures, but such lessons are learned in the months spent reflecting after a failure, not in the midst of it. Demanding absolute transparency at such an early stage for the sake of it makes people defensive, exacerbates fears, and ultimately makes organisations less likely to be transparent.

This isn’t to suggest that we shouldn’t aim to be completely transparent, we should, and we should hold ourselves accountable to the highest standard. But if we want others to take up the challenge, we must be sensitive to the fears that accompany increased transparency, and understand that the first steps are often the most daunting; they are certainly the most critical.

The ImpACT Coalition wants third sector leaders to make a genuine commitment to improve accountability and transparency and not just a token gesture (a ‘commitment’ to improve over a five year period holds no weight). Improving accountability and transparency should be driven by a genuine desire to communicate more effectively with stakeholders, not just because it looks good.

It is this cautious yet optimistic approach to accountability and transparency in the early stages that ultimately dictates its success, and the ImpACT Coalition exists to support organisations on this journey. Is it daunting? Yes. Is it challenging? Yes. It is worth it? Absolutely!