Choosing methods
Evaluation questions can be answered in a variety of ways. The choice of the method is therefore critical in order to get the most from the evaluation resources available. This is normally an operational choice that can be finalised only when the field of analysis has been reconstructed and there is enough information about the availability of data. However during the planning phase, it is necessary to make some choices. The choice of methods is influenced by:
- the reliability of the programme theory;
- the level of consensus between the stakeholders;
- the type of programme to be evaluated;
- the point in the programme cycle at which the evaluation takes place;
- the theme or sector of intervention of the programme.
The section of the GUIDE on choosing methods and techniques provides further information and guidance on the choice of methods. Sourcebook 2 elaborates specific methods and techniques, the Glossary provides definition of tools in less common usage.
The role of Guidance
The fact that Cohesion Policy evaluations are sometimes compulsory and that both the European Commission and National authorities issue guidelines about when and how to perform the evaluative exercises is a mixed blessing. On the one hand it can routinise the decision to evaluate. Evaluation can become an obligation to humour the above-mentioned institutions, with no clear added value for programme managers. On the other hand it can provide a much needed and welcome guidance both to the planning authorities and to the evaluation teams about the expected behaviours and results. Certainly the presumption that evaluations should be undertaken and the availability of guidance on their scope has been an important stimulus for the development of evaluation capacity as discussed further in Part 3.
Guidelines are especially useful to set the parameters for evaluation in relatively de-centralized programmes where it is important that common priorities, criteria and procedures are adopted. This can ensure a degree of comparability. Such guidelines have traditionally been developed by national or European authorities. They can also be useful if developed by programme managers or even evaluators when the overall evaluation process within a single programme is likely to be decentralized. In socio-economic development when participative evaluations and self-evaluations are common, some basic guidance, especially if priorities are developed collaboratively with local actors, can be most effective.
For the 2007-2013 programming period, the obligations for evaluation during the programming period have been reduced, in an attempt to move from a compliance-based approach to one based on needs. The Commission's guidance advises the establishment of a multi-annual evaluation plan to guide the evaluation process. This approach will require that programme authorities reflect more deeply than perhaps they have done in the past on what they want to evaluate and when. The Evalsed Guide should provide some additional support in this regard.
Key Resource decisions
Evaluations can also be seen as an attempt to second-guess programme managers' choices. More often than not they are under the impression that they already know most of the things that the evaluators tell them.
This is why it is important to involve the political authority, or at least the top management together with the most important external partners of the programme, in the planning of the evaluation. This does not mean involving them in the more technical decisions but making sure that they have the possibility to influence the following four fundamental questions:
- The reasons for the evaluation?
- Who is in charge of the overall exercise?
- How much to spend for the study?
- Who will perform the work?
Reasons for the evaluation?
This is the most fundamental question. As we have seen, there are different possible general purposes of an evaluation, there are different specific aims and there are different possible evaluation questions. Making sure that the choice reflected in the terms of reference is shared by top decision-makers and by the most relevant partners of the programme lends credibility to the whole exercise.
Who is in charge?
This involves:
- the members of the Steering Committee;
- those who write the terms of reference; and
- those who act as a liaison between the administration and the evaluation team.
Those in charge must be senior enough to have direct access to the policy makers in order to share with them the knowledge that the study will produce. They must also be conversant with the theoretical and methodological problems of evaluative research. This is essential in order to form their own judgements on the reliability of the product, as well as to pose the right questions to the evaluation team.
Ideally, therefore, the people in charge of the evaluation should have some experience of the practical work of evaluation, having done it in the past.
How much to spend?
It is difficult to decide how much to spend on an evaluation on an a priori basis. In general terms for large scale relatively routine programmes the budgets required for evaluation will be a small proportion of the programme resources (normally less than 1%). On the other hand for interventions that are relatively innovative and pilot in character and where evaluation has a strong learning and participatory aspect the costs are likely to be a relatively high proportion of programme (up to 10%). There are incidences where up to 5% of programme budgets have been devoted to evaluations that are effectively part of management's implementation strategy. For example, where evaluation includes a strong formative element intended to assist managers and stakeholders with their work.
The most appropriate basis for determining the budget is the nature and scope of the work required. Good evaluation requires inputs from good evaluators and the commitment of those commissioning the work and stakeholders alike.
In practice it is common in socio-economic programmes to spend sums unnecessarily when evaluations address routine topics but not to spend enough when programmes are innovative. This is, of course, the danger when evaluation is intended primarily for accountability or monitoring purposes.
Budgetary resources should not be a factor limiting the quality of an evaluation. However, there are diminishing returns. At the ex ante stage the time available to inform programme formulation and data availability are likely to be limited. At the mid term stage the size of beneficiary surveys and extent of stakeholder consultation will have a major influence on resource requirements. At the ex post stage the quality of on going monitoring and evaluations that have been undertaken rather than the budget per se is likely to be the main limiting factor.
Who performs the evaluation?
Should an evaluation be conducted by an external team or should it be conducted in house? There are advantages and disadvantages with either solution. External teams will often have greater specialist expertise and may be seen as independent, which can be important for the credibility of the evaluation. In-house evaluators will have greater familiarity with institutional and management requirements and may well have easier access to information and key personnel. They may, however, not be seen as independent and may lack specialist expertise. In part, this relates to decisions about capacity development within public administrations. Some have made a serious long-term commitment to in-house evaluation capacity located in specialist units. When these are clearly separated from operational management they can overcome concerns about their independence.
There are a number of structural approaches to ensuring the independence of in-house evaluators from programme managers. One approach is to locate the evaluation function in a separate organisational unit or division for example, in planning or strategy rather than in operations. Another is to ensure that higher levels of management separate from both operations and evaluation are explicitly involved in follow-up of evaluation recommendations and conclusions. This can act as a counter-balance to any tendency to ignore evaluation reports, for example, by holding all parties accountable for follow-up.
However, independence is not only a structural matter. Developing an ethos of independence among in-house evaluators (and supporting a similar ethos among external evaluators) can be an important way of ensuring behavioural independence. Furthermore, developing an evaluation culture in the relevant administrative units, one that is self-critical and open to new evidence and to ideas for improvement, can also strengthen the independence of the evaluation function.
There may be different logics appropriate for different stages of the evaluation and programme cycle. It may be preferable to rely more on internal resources for formative evaluation inputs or for ex-ante exercises but depend more on external resources for the ex-post evaluation of impacts and results.