Evaluating Donor-Level Results Measurement Systems

Donors have established a variety of systems for collecting and aggregating results information for aid-funded programs to assess whether funds are used efficiently for supporting desired outcomes. These results data enable aid organizations to assess which strategies and programs are effective, identify elements of programs associated with better results, demonstrate accountability to external stakeholders, and make decisions about allocating further funding. A variety of stakeholders have an interest in measuring aid results, and international agreements such as the Paris Declaration in 2005 and the subsequent Accra Agenda and Busan High Level Forum commit donors to improving aid effectiveness. Donors’ efforts to measure results vary, however, according to the purpose and audience for the results information they collect.

EPAR reviews the relative effectiveness of aid disbursed by bilateral and multilateral donors in a separate report.

Last year, the Evans School Policy Analysis and Research (EPAR) group at the University of Washington prepared a report reviewing bilateral and multilateral donor organizations’ approaches to results measurement and examining trends and gaps in how donors measure and report on their performance.

EPAR’s review evaluates donor-level approaches to measuring and evaluating aggregate results for 12 bilateral organizations and 10 multilateral organizations. The bilateral organizations comprise the primary aid agencies of selected Development Assistance Committee (DAC) member countries and represent 86.8% of total bilateral Official Development Assistance (ODA) in 2013 (Figure 1). The ten multilateral donors represent 74.7% of total multilateral ODA in 2013 (Figure 2).

Comparing Donor-Level Results Measurement Systems

Aid results information has multiple uses, including accountability to external stakeholders, institutional accountability and strategic management, and management and improvement of programs, projects, and policies. Figure 3 shows a hierarchy of performance measurement, from the least intensive measures, such as activities and outputs which can be directly tied to the organization, to more complicated measures, such as outcomes and effectiveness which require monitoring and evaluation over time. These types of measures serve different purposes, with spending information useful for accountability, output information useful to report specific organizational results, and outcome and effectiveness information useful for strategic management and program implementation.

Figure 3: Hierarchy of Performance Measurement

Donors also measure results at multiple levels, evaluating the performance of specific programs, recipients, countries, or portfolios that they fund. EPAR’s report focuses on results measurement at the donor-level by considering how 22 selected organizations evaluate the performance of their aid. The report analyzes each donor results measurement system using 78 indicators covering institutional design, level of evaluation, use of different types of measures (coordination and alignment, outputs and implementation, outcomes and impact, and costs and effectiveness), and reporting and use of results information. Data for the report include peer reviews of donor performance, donor evaluation plans and frameworks, and donor results and reporting documents.

Research Findings

EPAR finds that donors report several types of results and adopt varying approaches to results measurement. A common challenge, however, arises with aggregating results to the donor level, often stemming from funding and coordination limitations at the project, portfolio, country, and donor levels. EPAR does not find clear or consistent differences in approaches to results measurement among donors from different regions or between bilateral and multilateral donors, though some trends distinguish these groups of donors. For example, bilateral donors rate higher on coordinating with recipient-country systems, while multilateral donors are more explicit about the attention they give to measuring outputs and outcomes and are more likely to compare costs to outcomes to evaluate cost-effectiveness.

Three posts will follow in this series to further present the evidence and trends from EPAR’s research into donors’ results measurement systems. These posts will be organized around the questions:

1. What systems and processes do donors use for measuring aid effectiveness?
2. What types of donor-level results measures do donors use to evaluate their aid funding?
3. How do donors use the results data they collect?

These posts on donor-level results measurement systems follow a series of posts summarizing EPAR’s research on aid-recipient country monitoring and evaluation (M&E) systems, including their level of institutionalization and coordination, effectiveness of data collection, analysis, and use, and alignment and harmonization with development partner M&E systems.