Evidence-Based Policy Making, Policy Evaluation, and Effective Altruism: Considerations for a Research Project on Social Impact Measurement in Atlantic Canada

Philanthropy is not one of my usual research interests, but becoming a part of PhiLab has given me the opportunity to consider intersections between philanthropy, my own background in political science – specifically evidence-based policy making (EBPM) and policy evaluation – and my personal interest in effective altruism (EA), which essentially means evidence-based philanthropy. This article discusses and compares EBPM, policy evaluation, and EA. It concludes by considering some important implications for the philanthropic sector in Atlantic Canada. 

EBPM suggests that policy decisions should give consideration to the best evidence available, rather than using evidence selectively – or not at all – to support an ideological position; it is the opposite of opinion-based policy making (Davies 2004). Exactly what counts as evidence, and exactly how central its role should be, are matters of debate. Conceptualizations of EBPM run across a spectrum from tight EBPM – which privileges peer-reviewed academic research – to loose EBPM – which also considers stakeholder accounts, professional experience, public opinion, political know-how, and other inputs (Richards, 2017). Critiques caution that misuse or overuse of such definitions can suppress legitimate sources of information (Laforest and Orsini, 2005). We should also be wary of broad claims that a certain policy or system is evidence-based, which can ironically quash debate and mask processes ultimately rooted in a particular ideology or perpetuating a certain privilege (Packwood, 2002). Due to this debate, many scholars and practitioners are more comfortable with terms like evidence-influenced, evidence-informed, or evidence-aware policy making (e.g. Nutley, 2003; Julnes, 2007).

Policy evaluation is a type of practical and applied policy analysis to determine the impact and effectiveness of a given policy (Lacroix and Richards, 2015), which naturally can contribute to EBPM. It typically involves measuring the effects of a given policy and comparing them to the policy’s overall goals (Pal, 1989). Policy evaluation can also ask whether an alternative would have better met the goals, whether the goals themselves are relevant to the problem at hand, or whether there are some unintended effects worth considering (Howlett, Perl, and Ramesh, 2009). In summary, it can consider criteria of effectiveness, efficiency, and/or equity (Lacroix and Richards 2015). Policy evaluation can be conducted by scholars, think tanks, non-government organizations, or interest groups, but is most typically performed internally by governments themselves; it comprises the last of five theoretical stages in the typical policy making process – i.e. agenda setting, formulation, decision making, implementation, and evaluation (Howlett et al. 2009). Like EBPM, it also runs the risk of overly technical and rigid application, which may counterproductively disempower the relevant communities (Ife, 2016).

My familiarity with the above concepts comes from my scholarly training, but my exposure to EA comes from organizations such as GiveWell and the Centre for Effective Altruism. The latter defines EA as “using evidence and reason to figure out how to benefit others as much as possible, and taking action on that basis”. More specifically, EA is not just “doing what feels right” but using “evidence and careful analysis to find the very best causes to work on” as well as “following through” and “being generous with your time and money to do the most good you can”. In my opinion, “evidence” and “reason” in this regard include a broad range of qualitative or quantitative data, but EA organizations (e.g. GiveWell) tend to focus on quantitative measures and the metric of lives saved, leaving some debate on whether EA overlooks certain important causes in doing so (Pue, 2016).

Some of the emerging similarities between EBPM and EA, then, are:

• both use negative definitions (e.g. EBPM is not opinion-based policy making; EA is not just doing what feels right)
• both pursue maximum societal benefit using evidence, analysis, and reason
• both encounter similar critiques, especially over what should count as evidence and the dangers of misuse

To illustrate some more specific similarities, GiveWell’s process starts by identifying a priority area (i.e. the global poor) and then considers various charities in that area for their effectiveness, cost-effectiveness (i.e. efficiency), room for additional funding (i.e. equity), and transparency – the ease of acquiring information for assessing the other criteria. The recommended process of 80,000 Hours (a subsidiary organization of the Centre for Effective Altruism) also begins with a decision about a pressing global problem, but then assesses relevant charities by going a bit broader to consider organizational strength and team potential, and uses room for additional funding as a tie-breaker. The comparability of these procedures to policy evaluation above (e.g. the criteria of effectiveness, efficiency, and equity) is really quite stark.

The prevalence of such overlap suggests that some principles and processes can be adapted from the context of policy to the context of philanthropy, and vice versa. Just as there are similarities between evidence-based policy making and evidence-based philanthropy, there would be similarities between policy evaluation and philanthropy evaluation. The procedures followed by EA organizations in evaluating social causes may be adaptable to contribute to EBPM, and the tools of policy evaluation can probably be adapted to questions of philanthropic impact under EA; this network of related conceptual areas gives rise to a number of possibilities. See the table below for a summative comparison and grouping of roughly synonymous terms.

I hope to explore the practical implications of this rich conceptual overlap as part of an emerging research project with the Atlantic Hub. This project will partner with the Rural Communities Foundation of Nova Scotia to develop tools for evaluating the comparative benefits of their small grants, based primarily on social impact measurement training from the Community Sector Council of Nova Scotia. One key output of the project is to develop best practices and increase capacity for evaluating philanthropy in Atlantic Canada. This is primarily a question of philanthropy evaluation (i.e. the rightmost column in the above table), but it can be supported by the broader fields of EBPM, policy evaluation, and EA (i.e. the other three cells in the above table). These potential conceptual intersections are particularly important because Atlantic Canada’s philanthropic landscape is relatively poorly understood and may differ substantially from other regions in Canada; for example, charitable activity probably has more of a community focus and may involve more donations of time or goods instead of money (Reid, 2019).

For instance, some definitions of EA refer to both time and money (see above), and policy evaluation is meant to cover a variety of policy instruments, such as messaging, resources, and authority, in addition to money (Hood and Margetts, 2007). This breadth is suitable for the nature of philanthropic activity in Atlantic Canada. Moreover, the extensive range of policy evaluation may mean that it can offer particularly appropriate support for the philanthropic sector in Atlantic Canada. It acknowledges that evaluation:

• can be conducted by a variety of organizations, accommodating the smaller-scale groups found in Atlantic Canada
• can consider a diversity of data both quantitative and qualitative, including non-monetary inputs and non-statistical evidence that may be more readily accessible in Atlantic Canada
• can be conducted at any jurisdictional scale, including at the community level, which is important and appropriate for Atlantic Canada

Ultimately, my aim for this future research is to explore what the intersection of EBPM, policy evaluation, and EA can offer the philanthropic sector in Atlantic Canada, and vice versa. More broadly, I am also curious about whether comparable lessons can be extracted from other endeavours of “evaluation” generally, such as measuring the effectiveness of science outreach events.