This section more precisely discusses the various activities and issues that are needed when planning evaluation. In particular, it considers:
- Scoping and defining the object of evaluation;
- Identifying and involving stakeholders;
- Analysis of the programme theory and policy objectives, which underlie the interventions.
Defining the object of evaluation
The decision to evaluate is an opportunity to define limits in terms of institutional, temporal, sectoral and geographical dimensions. This is what is known as the scope of the evaluation or the "evaluation object". Defining the scope of an evaluation amounts to asking the question: What is going to be evaluated?
Evaluation scope can be specified in at least four respects:
- institutional (European, national or local level);
- temporal (time-period under consideration);
- sectoral (social, industrial, environmental, rural, etc.); and
- geographical (which part of the European territory, which region, town, nature reserve, etc.).
A programme is notionally delimited by finance and by the territory concerned and by the programming period. It is, however, useful to first consider:
- Is the intention to limit evaluation to the funding of the programme or to include other national, regional or local funding that is, to a greater or lesser degree, directly related to the programme?
- Is the intention to limit the evaluation to interventions in the eligible area or to extend observations to certain neighbouring areas that encounter similar development problems?
- Is the intention to limit the evaluation to funding allocated within the programming cycle under consideration or to a certain extent to include funding of preceding cycles?
- Is the intention to limit the evaluation to a part of the programme, thus allowing more indepth analysis?
It is normally helpful to adopt a relatively strict definition of the scope of the evaluation. Experience has shown that during the evaluation process stakeholders may wish to examine almost everything. In order to reach conclusions, the evaluation should be confined to an examination of the programme or a part of the programme and its most essential interdependencies with other public policies and interventions.
This risk of the scope widening is particularly great for ex ante evaluations. These can turn into exercises in forecasting or speculation that are far from the object of the evaluation. In ex ante evaluation it is best to limit the scope of the evaluation strictly to the programme proposals.
Commissioners of evaluation are often unwilling to restrict the scope of the evaluation questions they expect to cover. One contribution that evaluators can make is to identify those questions most central to programme success based on programme theories that identify intervention logics and implementation chains. Sometimes the best way to prioritise evaluation questions and focus the evaluation is to discuss practical constraints like time and resources.
For an evaluation to be useful, the decisions likely to be taken and which can be informed by the evaluation, must be stated as precisely as possible. Often commissioners, not wanting to influence the evaluation team too much, are reluctant to express in advance the changes they think should be made or their doubts concerning the effectiveness of a particular action. The intention is commendable: reveal nothing in advance to see whether the evaluation team reaches the same conclusions! Experience shows, however, that evaluation has little chance of documenting intended decisions if these are not known in advance by those who are going to collect and analyse data in the field. Socio-economic reality is highly complex and the evaluation team is confronted with a large number of observations and possibilities for making improvements. Verifying hypotheses which are in the event of little interest to officials, managers or other stakeholders is not realistic.
Identifying and involving stakeholders
As we have already seen socio-economic development includes several different types of projects, programmes and policies - this implies the number of actors or interested parties is often quite large. Evaluation experience suggests that this is far from being an obstacle to a good evaluation. On the contrary it offers opportunities that should be exploited in order to pose the most appropriate questions and give the most useful answers.
Activities on the ground impact on the number of stakeholders involved in policy making. In particular, the emphasis on the partnership principle is based on the view that the involvement of non-Governmental institutions and civil society actors will improve the quality of socio-economic development, both from the point of view of defining a comprehensive set of objectives and in terms of facilitating the implementation process.
Other factors which have reinforced the trend towards involvement of large and diverse groups of institutions and actors include the influence of vertical and horizontal partnerships, the emergence of multi-level governance and application of subsidiarity, the establishment of cross-cutting policy priorities such as sustainable development or equal opportunities and the recognition of the role played by social capital in socio-economic development.
The emergence of local and territorial development, where different policy sectors and sources of financing are integrated in an attempt to enhance the socio-economic development of an area, makes the identification of stakeholders and their involvement in the programme formulation process (the bottom up approach to planning) an essential step of the whole exercise.
Even in simpler programmes and projects there are always a number of actors whose interests are affected, positively or negatively, by the planned or implemented activity.
In all cases therefore, identifying the potentially affected actors (in ex ante evaluations) those actually affected (in mid term, interim or ex post exercises), and somehow involving them in the evaluation process is paramount to take into consideration points of view, indirect effects or unintended consequences that can be very significant for describing the effects, understanding the causality chains and judging the results.
The emphasis on the identification of stakeholders has so far been couched in terms of its practical benefits to understand the programme better, to ask better questions and to obtain good quality information. However, there is an additional rationale for identification and involvement of stakeholders. Evaluators along with programme managers have an interest in ensuring that there is ownership of evaluation findings. Only in this way is it likely that those involved will take evaluations seriously and act on recommendations or define their own action priorities on the basis of findings.
The first question that must be asked, after the scope of the evaluation has been defined, is therefore quite straightforward: Who are the individuals, the groups or the organisations who have an interest in the intervention to be evaluated and can be interested in the process or in the results of the evaluation itself? This phase in evaluation parlance is called the identification of the stakeholders.
Establishment of a steering group and terms of reference
Ideally this exercise should take place before defining the details of the evaluation to be performed: by taking into consideration their points of view, it is possible to decide the most relevant questions that should be answered. But when this is not possible, for instance because it has not been possible to identify all the interested parties at an early stage, some sort of involvement is desirable.
The second question that should be asked is: How is it possible to make sure that the stakeholders provide the relevant inputs to the design, management or content of the evaluative exercise?
The involvement of the stakeholders can take place at very different levels
- At a minimum the evaluators should make sure that stakeholders provide evidence (data, information, judgements, etc.) as part of the evaluation process. Many methods and techniques described in Sourcebook 2 can be used for this purpose: individual interviews, surveys, focus groups, etc.
- At the other end of the continuum, the stakeholders can be involved in steering the whole study, including defining priorities, evaluation questions and associated criteria. Often this means involvement in the Steering Committee for the evaluation project, as we will see when we discuss the management of the evaluation process.
In practice the involvement of stakeholders in most programmes falls somewhat in the middle. If the participation of stakeholders in the Steering Committee is restricted to the official institutional and social partners, some way to provide feedback to other actors that are able to provide information and judgements is widely practised through the dissemination of reports, ad hoc meetings and similar instruments.
Programme theories and logic models
The stakeholder consultation phase also provides an opportunity to reconstruct the logic of the programme prior to its launch. As we have seen in Part 1 of the Guide, there are different and often competing theories underpinning interventions.
Ideally, every programme or policy would state clearly the set of assumptions on the basis of which the desired goal in our case socio-economic development, can be reached through the resources allocated and the interventions funded. These assumptions whould be consistent with each other and would be supported by evidence. This is rarely the case in practice, especially in the complex world of socio-economic development.
A further step in evaluation planning, therefore, is to reconstruct the programme theory underpinning the object of evaluation. This is mainly to assess the ability of the programme to reach its intended goals, (i.e., development). A clear identification of the reasons why this should be expected is an important precondition to posing the right evaluation questions.
This emphasises how programming and evaluation are interrelated. Programme managers and planners need to be aware that there are tools available that can help reconstruct the chain that links the general goals of the programme, the specific intermediate objectives, the activities put in place by the implementers and finally the results and the consequences of these activities. The choosing methods and techniques section of the GUIDE (and Sourcebook 2) discusses and provides examples of various tools and techniques, ( logic models, log frames, programme theory, theory of change) that can assist in the reconstruction of programme intervention logics and implementation chains.
In conjunction with stakeholder consultation and analysis, the application of these methods can help to pinpoint the possible critical aspects of the programme implementation and therefore to focus the evaluation appropriately.