Another set of considerations that need to inform the selection of methods and techniques is the different evaluation purposes that were identified earlier:
- Planning/efficiency - ensuring that there is a justification for a policy/programme and that resources are efficiently deployed.
- Accountability - demonstrating how far a programme has achieved its objectives and how well it has used its resources.
- Implementation - improving the performance of programmes and the effectiveness of how they are delivered and managed.
- Knowledge production - increasing our understanding of what works in what circumstances and how different measures and interventions can be made more effective.
- Institutional and network strengthening - improving and developing capacity among programme participants and their networks and institutions.
To an extent, particular methods and techniques are associated with these different purposes. For example:
- With regard to planning and efficiency, methods are primarily concerned with resource allocation and economic efficiency. Various forms of impact analysis will be appropriate, as will different forms of cost-benefit analysis. In broader managerial terms, objective-driven techniques such as those characteristic of some logical framework approaches will also be used. There are a range of such methods and techniques described in Sourcebook 2. These would include, for example,input-output analysis and efficiency analysis.
- With regard to accountability, methods are primarily about judging performance against some standard or target and applying relevant criteria for success and performance. In its most straightforward form, this is close to what is classically the work of auditors. Comparisons against standards can be achieved in a number of ways. For example, indicators can be used to compare actual outcomes with expectations. Comparisons can also be made with external examples through benchmarking. Where there is no easy way to compare externally, as is often the case in the context-specific world of socio-economic development, comparisons may be made on a before and after basis showing changes over time. In general the evaluations that are largely about accountability will tend to emphasise financial and monetary measures and quantitative techniques. However, this is not always so, as policy makers often find it helpful to have illustrative case material and qualitative descriptions of development outcomes to support more abstract descriptions in terms of finance or money alone.
- With regard to implementation, typical methods will attempt to describe processes and interim outcomes, in order to provide feedback to those responsible for programme implementation. Many of these methods and techniques will be informed by an organisational and policy studies background. There may be comparisons made between the performance of different administrative units, for example, are different regions or municipalities making more or less progress? Case studies of organisational and partnership arrangements will help understand the strengths and weaknesses of different implementation approaches. Often these kinds of methods will involve what are called formative evaluation methods and techniques. These place a particular onus on the evaluator to provide feedback in ways that will be useful and will help programme managers translate emerging evidence into practical action.
- With regard to knowledge production, methods will be closest to those used by academic researchers. They will be subject to demands for rigour, representativeness and the cautious interpretation of findings, especially where these may be inconsistent. Typically, for knowledge production purposes, evaluators will want to answer the question, what works? From a positivist perspective, this would be an area where experimental methods are seen as relevant. However, the diverse and bottom-up nature of socio-economic interventions, the way these are combined in particular configurations and the different localities and contexts where programmes take place, makes traditional experiments challenging to apply. For that reason that realist thinking, with its emphasis on the influence of context on outcomes, has become more common in these kinds of evaluations. Here the more complex question is asked: what works, for whom, how and in what circumstances? Methods and techniques suitable for this will generally involve comparison between different cases selected to demonstrate alternative interventions and alternative contexts. Such comparisons may be based on case studies, data-bases that structure intervention/outcome/context configurations or a range of other techniques that are able to capture and describe these different aspects of socio-economic development.
It is widely accepted in the evaluation community that reliable knowledge rarely comes from a single evaluation. For this reason there is growing interest in undertaking synthesis studies and various kinds of meta-analysis that try to build up what is known from as a large a number of evaluations as are available. As knowledge production has become more important with the commitment of policy makers to evidence-based policy-making, various kinds of meta-analysis have become widespread. This form of analysis is strengthened if, when designing evaluations that might subsequently be the included in meta-analyses, some standard structures and data items are collected across all cases.
With regard to institutional and network strengthening, it is now widely recognised that evaluations are not exclusively to meet the needs of programme managers and sponsors but also have to be owned by a wide group of stakeholders. Furthermore, the effective delivery of programmes often depends on the capacities of the institutions and organisations from which these stakeholders come, as well as broader civil society networks. Very often the methods that would be appropriate in these settings will be participatory: placing an emphasis on close collaborative work between the evaluators and the institutions and networks involved. These participatory approaches will not only be important in formulating evaluation questions but also when generating data and using these results of evaluations. For example, in a community setting where there are many interests and perhaps a lack of a shared view, evaluators may need to work with community representatives to develop consensus if the results of an evaluation are to be used. Of course, approaches to institutional and network strengthening can be pursued in a much more direct way. For example, studies may be undertaken of the administrative capacity of particular partner organisations in order to help them adopt more suitable management processes and information systems.