Donors have established a variety of systems for collecting and aggregating results information for aid-funded programs to assess whether funds are used efficiently for supporting desired outcomes. These results data enable aid organizations to assess which strategies and programs are effective, identify elements of programs associated with better results, demonstrate accountability to external stakeholders, and make decisions about allocating further funding. A variety of stakeholders have an interest in measuring aid results, and international agreements such as the Paris Declaration in 2005 and the subsequent Accra Agenda and Busan High Level Forum commit donors to improving aid effectiveness. Donors’ efforts to measure results vary, however, according to the purpose and audience for the results information they collect.
Last year, the Evans School Policy Analysis and Research (EPAR) group at the University of Washington prepared a report reviewing bilateral and multilateral donor organizations’ approaches to results measurement and examining trends and gaps in how donors measure and report on their performance.
EPAR’s review evaluates donor-level approaches to measuring and evaluating aggregate results for 12 bilateral organizations and 10 multilateral organizations. The bilateral organizations comprise the primary aid agencies of selected Development Assistance Committee (DAC) member countries and represent 86.8% of total bilateral Official Development Assistance (ODA) in 2013 (Figure 1). The ten multilateral donors represent 74.7% of total multilateral ODA in 2013 (Figure 2).
Comparing Donor-Level Results Measurement Systems
Aid results information has multiple uses, including accountability to external stakeholders, institutional accountability and strategic management, and management and improvement of programs, projects, and policies. Figure 3 shows a hierarchy of performance measurement, from the least intensive measures, such as activities and outputs which can be directly tied to the organization, to more complicated measures, such as outcomes and effectiveness which require monitoring and evaluation over time. These types of measures serve different purposes, with spending information useful for accountability, output information useful to report specific organizational results, and outcome and effectiveness information useful for strategic management and program implementation.
Figure 3: Hierarchy of Performance Measurement
Donors also measure results at multiple levels, evaluating the performance of specific programs, recipients, countries, or portfolios that they fund. EPAR’s report focuses on results measurement at the donor-level by considering how 22 selected organizations evaluate the performance of their aid. The report analyzes each donor results measurement system using 78 indicators covering institutional design, level of evaluation, use of different types of measures (coordination and alignment, outputs and implementation, outcomes and impact, and costs and effectiveness), and reporting and use of results information. Data for the report include peer reviews of donor performance, donor evaluation plans and frameworks, and donor results and reporting documents.
EPAR finds that donors report several types of results and adopt varying approaches to results measurement. A common challenge, however, arises with aggregating results to the donor level, often stemming from funding and coordination limitations at the project, portfolio, country, and donor levels. EPAR does not find clear or consistent differences in approaches to results measurement among donors from different regions or between bilateral and multilateral donors, though some trends distinguish these groups of donors. For example, bilateral donors rate higher on coordinating with recipient-country systems, while multilateral donors are more explicit about the attention they give to measuring outputs and outcomes and are more likely to compare costs to outcomes to evaluate cost-effectiveness.
Three posts will follow in this series to further present the evidence and trends from EPAR’s research into donors’ results measurement systems. These posts will be organized around the questions:
- What systems and processes do donors use for measuring aid effectiveness?
- What types of donor-level results measures do donors use to evaluate their aid funding?
- How do donors use the results data they collect?
These posts on donor-level results measurement systems follow a series of posts summarizing EPAR’s research on aid-recipient country monitoring and evaluation (M&E) systems, including their level of institutionalization and coordination, effectiveness of data collection, analysis, and use, and alignment and harmonization with development partner M&E systems.
If an organization with an existing culture of learning and adaptation gets lucky, and an innovative funding opportunity appears, the result can be a perfect storm for changing everything. The Results Data Initiative was that perfect storm for DG. RDI confirmed that simply building technology and supplying data is not enough to ensure data is actually used. It also allowed us to test our assumptions and develop new solutions, methodologies & approaches to more effectively implement our work.
From our experience understanding data use, the primary obstacle to measuring and organizational learning from feminist outcomes is that development actors do not always capture gender data systematically. What can be done to change that?
Development Gateway’s mission is to support the use of data, technology, and evidence to create more effective and responsive institutions. We envision a world where institutions listen and respond to the needs of their constituents; are accountable; and are efficient in targeting and delivering services that improve lives. Since late 2018, we’ve been operating under