- Evaluation competence should be brought in early by programme planners. In particular, this can help to clarify objectives of programmes and interventions. This activity, although employing evaluation competence, is quite separate from mainstream evaluation activities. It needs to occur at the design and planning stage. However, this can make subsequent evaluation easier and more successful. Various techniques such as evaluability assessment and preparing an analysis of programme theory can be deployed for this purpose. In general, in order to ensure independence of the evaluation proper, it would be best to use different evaluation teams or resources for this programme planning work than for the main evaluation.
- A similar evaluability assessment should be undertaken by evaluators when they begin their work. To some extent this may overlap or repeat what has already taken place at planning stage. However, the purpose here is different. It is to ensure that a feasible evaluation plan is produced and to clarify how evaluation outputs will be used. This is consistent with a general expectation of evaluators that they should be concerned with how their results, conclusions and recommendations are used from the earliest possible stage of their work.
- Stakeholders, programme managers and policy makers, potential beneficiaries and partners should be involved in the evaluation from the earliest stages, where practicable. This will ensure that the evaluation design and plan will include their priorities and agendas. It will also ensure that they feel some sense of ownership of the outputs of the evaluation and are more likely, therefore, to find these useful and use them. On the other hand, it may be necessary to be selective in deciding on which voices finally determine the evaluation agenda, in order to retain focus and ensure the evaluation is manageable. Overarching priorities should be shaped by the intentions and logic of the programme or intervention whilst remaining open to unintended consequences especially for intended beneficiaries.
- Evaluations need to be actively but sensitively managed. This will ensure that commissioners are aware of choices that need to be made along the way. It will also ensure that evaluators receive sufficient support, access to information and briefing as to changes in policy and context. Those responsible for commissioning an evaluation and programme managers are the most suitable people to manage the evaluation because they will be aware of its background and rationale.
- It is usual to derive criteria for an evaluation, i.e., judgements as to the basis for positive and negative assessments of progress, from the objectives of a programme. It is also important to include a wider set of criteria that derive from social needs. For example, is this programme useful and helping those for whom it is intended? Does it support equity or not? Is the programme consistent with other policy initiatives? And is it delivered in an efficient and legitimate way? Maintaining this broader perspective ensures that for part of their work at least, evaluators are able to stand outside the logic of the programme and take a critical perspective on what it is trying to achieve and how it does it.
- The importance of evaluation questions in an evaluation design cannot be overstated. The temptation otherwise is to gather large quantities of data and produce sometimes technically sophisticated indicators which make little contribution to practice or policy. There is, of course, a problem formulating the evaluation questions in a way that they are likely to be able to be answered. While this is a technical question and this part of the GUIDE has offered suggestions about how to formulate questions appropriately, there is here also the overarching concern for use. You should try to ask questions that someone will find useful. However, use should not itself be defined too narrowly. We are talking here not just about the instrumental use of evaluation by managers. We are also talking of uses that citizens and civil society groups may make of evaluation in support of democratic processes and accountability.
- We have specified in some detail the content and form of ideal Terms of Reference for an evaluation. This is part of the general question of design and the choices that can be made at the design stage which can influence the quality and direction of an entire evaluation. It is important therefore not to simply follow a standard framework with pre-drafted paragraphs. Rather it should be recognised that defining scope, clarifying the users of the evaluation and deciding the skills required for an evaluation team, are among the most important decisions that are made during the course of an evaluation.
- It used to be common to regard the use of evaluation as being confined to acting on recommendations and final reports. It is now understood that evaluation use can be supported and occurs throughout an evaluation. So-called process or dialogue use should involve stakeholders in evaluation thinking from the beginning. There are evaluations where the conclusions and recommendations are rejected but stakeholders, especially the core stakeholders involved in the steering committee, nonetheless find the evaluation useful. It can help them to clarify their own thinking and understanding and spark off innovative ideas for improvements. This continuous process of communication provides a particular context for the dissemination of evaluation reports and findings. Promoting dialogue during the course of an evaluation is likely to ensure that when stakeholders receive reports they will be better prepared and receptive.
- It is often easier for programme managers and those who commission an evaluation to confine judgements of evaluation quality to the outputs in reports of the evaluation itself. However, this quality control process provides few opportunities for learning and improvement in the way the evaluation itself is managed. A quality assurance perspective of the kind that has been advocated in this part of the GUIDE provides a context in which to explain the strengths and weaknesses of evaluation outputs. It also offers an opportunity for those who commission evaluations to learn how to improve evaluations in future.
- Consideration should be given at an early stage to how evaluation findings will be put to use. Some use will stem directly from the findings and recommendations of the work. Evaluations can also be helpfully combined with other evidence to inform debates. The process of evaluation can bring benefits in terms of structuring inquiry and institutional reflection. Close attention to the factors that influence the use of evaluation work will maximise its contribution.