In the section of the GUIDE on evaluation and socio economic development we have emphasised how socio-economic development, not being a precise science, is complex and uncertain, if only because there are often several, and not necessarily mutually compatible, theories that each support different development strategies.
Planning documents are first and foremost an essential part of the planning and project cycle, and as such a fundamental input in the policy for socio-economic development. They are, however, also policy documents, that usually have to be agreed by many actors, from different territorial levels, and with very different values, aims and priorities. It is not surprising, therefore, that these documents are often vague, that they try to cover every possible angle of the problem in a sometimes generic way (even if this implies spreading thinly the available resources) and that some of the objectives that they identify are mutually contradictory.
This danger is most present when the complexity of the programmes increases, as with the new generation of socio-economic development policies, stressing the territorial approach, the emphasis on sustainability, the need for extended partnerships and for various mainstreaming principles.
According to one traditional vision, these uncertainties make the task of evaluating the results of socio-economic development programmes difficult, if not impossible. Without clear goals, a coherent intervention theory, and a precise programme design - it is assumed - the identification of what to evaluate and of the criteria for evaluation becomes arbitrary and subjective. Whatever the merits of this received wisdom, there is also another way of looking at the problem. It is exactly because of the existence of multiple objectives and complex programmes that evaluation becomes essential.
Ideally, these evaluation concerns should be taken into account in the programme formulation phase, and this should help to prevent problems, such as conflicting objectives. Conceptualising the expected results in operational and therefore measurable terms (e.g., by building from an early stage a monitoring and indicator system) is a very powerful means of helping decision-makers to formulate better programmes. For this reason, involving the evaluation and the evaluators as early as possible is a prerequisite for a good socio-economic development programme
Importantly, many evaluation techniques (such as evaluability assessment, constructing a programme theory and SWOT analysis) can be used, from a very early stage, in order to clarify the starting priorities and the logic that underpins programme interventions.
Using evaluative techniques and adopting an evaluative approach from the very beginning will help stakeholders to develop a common language, while the identification of some tangible and measurable intermediate results will help the general framing of the implementation process. This also allows for milestones to be established that can ensure that the process is kept on track throughout the lifetime of the programmes.
But even after the planning documents have been finalised, the contribution of evaluation to programming can be very important. At this stage evaluation can help to make sense out of a confused collection of aims and a generic shopping list of possible projects. The use of the so called logic models, helps map the interests and policy priorities of the different stakeholders. If such models are developed in interaction with evaluators and policy makers, they may even lead to restructuring the programme. This is probably the most important contribution of evaluation to the programming phase.
It is important to note a very important and too often forgotten truth: the value of the evaluation for policy makers lies as much in posing the right questions as in providing precise answers. (This is discussed in more detail below). Sound methodology and reliable data are very important and will yield good answers to evaluation questions. However, ultimately the added value of evaluation for decision makers consists in facing them with questions, such as: What were the objectives of the exercise? Is this the equilibrium between the different goals that we really want? Can this level of performance be considered satisfactory?
Nonetheless, creating the conditions where evaluation questions can be answered as precisely as possible remains an important goal. In this context, the more precise the programme, the more explicit the potential trade-offs and synergies between the different goals and objectives, the more stringent the programme logic, the more reliable the programme theory, (i.e., the causal links between the projected actions and the expected results), the more comprehensive the indicators system, the more coherent the sequence of the intermediate results and the implementation process put in place, the easier and more precise will be the evaluation. Such an ideal situation will make the judgements about the success or failure of the programme sound, reliable and useful. It will increase accountability to the governance system and develop a better understanding of the ways in which the general goal of sustainable socio-economic development can be attained.
From the perspective of a prospective evaluator there is benefit where:
- The aims (or goals) are as explicit as possible and the concepts referred to are defined and commonly understood.
- Objectives are linked to interventions (or groups of interventions) and measurable outcomes.
- If interventions have multiple objectives, some explicit weight is attached to each objective.
- The objectives incorporate targets the achievement of which can be measured. The targets should have explicit timescales and should have been the result of negotiation between policy makers and those responsible for implementation.
- There are incentives that unambiguously encourage the achievement of objectives.
These conditions help the evaluation process to define criteria against which to judge whether an intervention might, is en route to or has achieved a successful outcome. A further condition which could support rigorous impact evaluation is reflection at the programme design stage on how the conterfactual can be tracked, i.e., what would have happened in the absence of the programme.
In practice these conditions are uncommon and as is elaborated below, the evaluation process itself must often contribute to the defining of objectives and criteria that reflect the ambitions that were set for the interventions
The Europeal Commission's recommended good practice approach in programming of evaluations