Valuing evaluation in public philanthropic foundations

The term ‘evaluation’ is used easily and frequently in philanthropic circles, but often with varied understandings of what it is and how to do it.  Along with its cousins, ‘impact’ and ‘performance measurement’, evaluation is seen as definitively good, but to be approached with caution.  Evaluation buttresses grantmaking programs; it provides evidence and confidence in doing the right thing.  Yet it’s expensive, in both dollar terms and time.  And what do we really do with it, anyway?

Building from this understanding of evaluation as ‘tricky but probably worth it’, we sought to understand how the senior leaders of public philanthropic foundations relate to evaluation.  We considered previous research specific to philanthropic foundations and evaluation (Anheier & Leat, 2018; Kinarsky & Christie, 2021; Leat, 2006, 2016; Williamson et al., 2017).

We were interested in all forms and subjects of evaluation in foundations.  The most common form of evaluation is asking (and hopefully also funding) the foundation’s beneficiaries to evaluate the work they completed with the grant funding. In addition to this type of grant evaluation, there is also an evaluative component within assessing applications for grants.  And evaluation can encompass the foundation’s own work in making those assessments and considering its own performance against mission.

Thus, although the focus of philanthropic evaluation literature is mainly on ‘how evaluation is done’, the focus of our article (Williamson & Kingston, 2021) is instead on ‘how evaluation is understood’.  Of particular interest to us was learning how evaluation is understood by senior leaders of public philanthropic foundations, those working ‘on-the-ground’ so to speak, rather than those who specialise in evaluation or are external evaluation consultants. The limited specialist evaluation knowledge of senior leaders makes their understandings of evaluation rich in idiosyncratic organisational culture and operational processes.

Our article details perspectives from interviews with 28 managers and trustees of Public Ancillary Funds (PubAFs) as part of a wider study on the accountability of public foundations. PubAFs must fundraise from the public, and donations to them are deductible against the taxable income of the donor, resulting in a significant element of accountability to the public for their effectiveness and evaluation of the distribution of their funds.  We chose to speak with both managers and trustees as we wanted both operational and monitoring perspectives, and because these two roles may in fact be filled by the same individual in smaller foundations.

Four main themes emerged through the exploration of how evaluation is understood from the perspective of these senior foundation leaders: motivations, values, criteria and processes.

• Leaders’ motivations for engaging in evaluation were mostly instrumental but were also about a sense of responsibility for the foundation’s resources.
• Leaders’ values influencing engagement in evaluation comprised both integrity and respect, and specifically included trust, honesty, identity and privilege.
• Leader’s perceptions on evaluation criteria prioritised the usefulness of the assessment for decision making. Perceptions focused on the quality and effectiveness of applicant and beneficiary organisations and projects, including their financial status, and regulatory compliance.
• Evaluation processes used by leaders were considered for their involvement of stakeholder groups, their utility and effectiveness, and their fit for purpose, particularly regarding financial cost and benefit. Here leaders also highlighted the value of reflecting on past successes and failures as an evaluative process.

Hearing the perspectives of senior leaders on evaluation has allowed us to build a richer understanding of how evaluation is understood and what it means in practice, within public philanthropic foundations. Practical implications of the research include a greater awareness of the multiple ways in which evaluative roles are understood in public foundations.  While this pluralism of understanding around evaluation in foundations may have a blurring effect, this study nevertheless finds that while evaluation as an activity may be debated, evaluation as a concept is more readily understood by senior foundation leaders. A key finding was that evaluative thinking strengthened the trustworthiness, honesty, and identity of PubAFs, supporting the importance leaders placed on evaluation in fostering donor confidence and enhancing organisational effectiveness.

One negative aspect of the findings was the external focus of much evaluation activity with a clear focus on donors as key, powerful stakeholders.  Much less attention was given to beneficiaries of grants, and the benefits or costs of evaluation to them as a crucial and genuine partner in delivering on a foundation’s mission (Kingston et al., 2019).  The scope of this study did not include donors’ and beneficiaries’ perspectives, and these should be a focus for future research.

By understanding what philanthropic leaders see as important within the evaluation culture and operating environments of their foundations, our findings may help to further the contribution of their work and subsequently enhance organisational practice.

This article was written by Alexandra Williamson, member of PhiLab’s International Research Unit in Australia, and Kylie L. Kingston.