What is about the purpose?
There are technical limitations that make it impossible to answer multiple questions or, more precisely, to provide quality answers to an excessive number of questions. This guide recommends a maximum of ten questions.
How to choose the questions
A first version of the evaluation questions is proposed on the basis of:
- The analysis of the intervention logic.
- The analysis of the intervention rationale.
- Issues that justified the decision to launch the evaluation.
- Issues to be studied, as stated in the terms of reference.
- Questions raised in the ex ante evaluation, where relevant.
In a second version, the list and wording of the evaluation questions also take into account:
- Issues raised by key informants at the start of the evaluation.
- Expectations of the members of the reference group.
Assess the potential usefulness of answers
Assuming that a question will be properly answered, it is necessary to assess the potential usefulness of the answer, by considering the following points:
- Who is to use the answer?
- What is the expected use: knowledge, negotiation, decision-making, communication?
- Will the answer arrive in time to be used?
- Is the answer not already known?
- Is there not another study (audit, review) underway, likely to provide the answer?
If the choice of questions has to be discussed in a meeting, it may be useful to classify them in three categories of potential utility: higher, medium, lower.
Check that nothing important has been overlooked
Experience has shown that it is most harmful to the quality of the evaluation if the following type of question is left out:
- Questions on efficiency and sustainability.
- Questions concerning negative effects, especially if those effects concern underprivileged groups.
- Questions concerning very long-term effects.
Assess the feasibility of questions
The feasibility (evaluability) of a question should be examined, but always after its usefulness. For this purpose the following should be consulted:
- The service managing the intervention.
- One or more experts in the field.
- One or more evaluation professionals.
If the choice of questions has to be discussed in a meeting, it may be useful to classify them in three categories:
- Strong probability of obtaining a quality answer.
- Average probability.
- Weak probability.
If a question is potentially very useful but difficult to answer, check whether a similar question would not be easier and equally useful. For example, if a question concerns a relatively distant or global impact, its feasibility could probably be improved by focusing on the immediately preceding impact in the intervention logic.
Discuss the choice of questions
The choice of questions is discussed at the inception meeting.
The selection is more likely to be successful if potential users have been consulted and have agreed on the selected questions, and if no legitimate point of view has been censored.