Custom Assessment Landscaping Methodology (CALM)

Background

Despite their importance, development data systems suffer from high levels of inefficiency, ineffectiveness, and redundancy. In the United States, United Kingdom, and New Zealand, an estimated 60-80% of public-sector technology projects experience significant disruptions or problems of some kind. Why is there such a high rate of failure? In part, because development data systems – particularly M&E systems – are often designed based on what data must be recorded and reported for accountability purposes, not what data is used to maximize learning. While accountability is critical for development programming, accountability at the expense of learning undermines the potential for gains in efficiency and effectiveness over time.

In order for data to be useful and used – and for programming to meet its policy objectives – agencies and organizations must find the proper balance between accountability and learning. Achieving this balance requires understanding the relationship between decision space and data use; and identifying what tools or processes can facilitate both accountability and learning objectives.

The CALM Approach

Achieving data use requires changing how data systems are implemented, which requires changing the traditional approach to designing data interventions. The Custom Assessment and Landscaping Methodology is one approach for developing tools and processes that increase the utility and impact of data for decision-making.

Operationally, Development Gateway’s CALM approach includes:
1. Document review of budgets, reports, and strategies to assess policies and resources
2. Results and programmatic data review to assess frequency, standardization, availability, and appropriateness of data (dis)aggregation
3. Interviews with agency staff and external partners to explore decision-making processes, de facto policy implementation, and related data needs
4. Technical assessment of existing systems for data collection, reporting, sharing, and use
5. Implementation recommendations on tools, processes, and/or strategic objectives to support institutional priorities

For both new and established data systems, CALM provides a decision space-sensitive framework for assessing user needs and developing interventions to balance accountability and learning. By first understanding agency priorities and user needs, the levers of technology, capacity, and strategy can be better implemented to inform organizational learning and decision-making.

More information about the methodology and DG projects and programs that used the CALM approach can be found in our white paper.