Most frequent biases
This risk is a threat to all data collection approaches. It results from a tendency to seek out evidence that is consistent with the intervention logic, rather than evidence that could disprove it.
When subject to this bias, the evaluation team and informants tend to focus on intended effects and systematically to overlook external factors, unintended effects, negative effects, interactions with other policies, outside stakeholders, alternative implementation options, etc.
This bias is avoided by relying on independent and professional evaluators.
In some instances, informants may be reluctant to freely answer questions, simply because they feel at risk. They tend to rigidly express the views of their institution or their hierarchy.
This bias is combated by guaranteeing confidentiality and anonymity in the treatment of answers. The interviewer should also insist on factual questions and avoid collecting opinions.
Those who have stakes in the intervention may distort the information they provide, with the aim of obtaining evaluation conclusions closer to their views.
This bias will be reduced if the whole range of stakeholders is included in the data collection work plan and if various sources of information are cross-checked.
This bias may be a matter of concern if the evaluation team generates quantitative data through a questionnaire survey. It should also be considered when using secondary data obtained from a questionnaire survey.
In this instance, the evaluation team should verify that the sample of surveyed informants is large enough and representative of the population as a whole.
Question induced answers
This bias and the following ones are frequent in interviews and questionnaires.
The way in which questions are asked by interviewers or the interviewer's reaction to answers can generate a bias which is either positive or negative.
Even the order of the questions in a questionnaire may change the substance of the answers.
This bias will be limited by having questionnaires designed and tested by experienced professionals.
Interviewees may not have a pre-determined opinion about the questions put to them. They try to make up their mind in a few seconds when responding to the interviewer or to the questionnaire. While doing so, they may be strongly influenced by the context.
Especially in the case of interviews, the evaluation team has to create a friendly (empathetic) atmosphere, at least for the sake of achieving a high rate of answers and fast completion of the survey.
The combination of the two introduces a systematic positive bias in the answers, which tends to overestimate the benefits of the intervention and to underestimate the role of external factors.
This bias is prevented by relying on properly trained interviewers.
Sample selection bias
People who agree to be interviewed may not be representative of the overall target audience.
This bias could be controlled by undertaking a special qualitative survey on a few "non-respondents", although this exercise brings additional costs.