IMPORTANT LEGAL NOTICE - The information on this site is subject todisclaimercopyright notice
Evaluation
  EUROPA > European Commission > EuropeAid > Evaluation > Methodology > Basics > How?
Last updated: 13/01/2006
EuropeAid

Methodological bases
Evaluation process (How?)
Data collection

EuropeAid
 

Reliability of collected data

 


Methodology
• Evaluation Guidelines
• Methodological bases
• Evaluation tools
• Examples
• Glossary
• Sitemap

Methods
• What
• When
• Why
• Who
• How

How
• Overview
• Strategy
• Questions
• References
• Design
• Data Collections
• Analysis
• Judgment
• Quality assurance

Data collection
• Overview
• Existing data
• Frequent difficulties
• Reliability

 
 
What are the risks?

While gathering information, the evaluation team faces various risks of biases which may undermine the reliability of collected data.


Why should biases be considered carefully?
  • For improving the reliability of data collection
  • For assessing the quality of the evaluation
  • For understanding the limitations of conclusions which draw on unreliable data
Top

Most frequent biases

Confirmation bias

This risk is a threat to all data collection approaches. It results from a tendency to seek out evidence that is consistent with the intervention logic, rather than evidence that could disprove it.

When subject to this bias, the evaluation team and informants tend to focus on intended effects and systematically to overlook external factors, unintended effects, negative effects, interactions with other policies, outside stakeholders, alternative implementation options, etc.

This bias is avoided by relying on independent and professional evaluators.


Self-censorship

In some instances, informants may be reluctant to freely answer questions, simply because they feel at risk. They tend to rigidly express the views of their institution or their hierarchy.

This bias is combated by guaranteeing confidentiality and anonymity in the treatment of answers. The interviewer should also insist on factual questions and avoid collecting opinions.


Informants' strategy

Those who have stakes in the intervention may distort the information they provide, with the aim of obtaining evaluation conclusions closer to their views.

This bias will be reduced if the whole range of stakeholders is included in the data collection work plan and if various sources of information are cross-checked.


Unrepresentative sample

This bias may be a matter of concern if the evaluation team generates quantitative data through a questionnaire survey. It should also be considered when using secondary data obtained from a questionnaire survey.

In this instance, the evaluation team should verify that the sample of surveyed informants is large enough and representative of the population as a whole.


Question induced answers

This bias and the following ones are frequent in interviews and questionnaires.

The way in which questions are asked by interviewers or the interviewer's reaction to answers can generate a bias which is either positive or negative.

Even the order of the questions in a questionnaire may change the substance of the answers.

This bias will be limited by having questionnaires designed and tested by experienced professionals.


Empathy bias

Interviewees may not have a pre-determined opinion about the questions put to them. They try to make up their mind in a few seconds when responding to the interviewer or to the questionnaire. While doing so, they may be strongly influenced by the context.

Especially in the case of interviews, the evaluation team has to create a friendly (empathetic) atmosphere, at least for the sake of achieving a high rate of answers and fast completion of the survey.

The combination of the two introduces a systematic positive bias in the answers, which tends to overestimate the benefits of the intervention and to underestimate the role of external factors.

This bias is prevented by relying on properly trained interviewers.


Sample selection bias

People who agree to be interviewed may not be representative of the overall target audience.

This bias could be controlled by undertaking a special qualitative survey on a few "non-respondents", although this exercise brings additional costs.


Top

Recommendations
  • Rely on an independent and professional evaluation team in order to limit confirmation biases.
  • Systematically mix positive and negative questions in order to reduce empathy bias and question bias.
  • Be highly credible when promising confidentiality and anonymity in order to limit respondents' self-censorship - and keep such promises strictly.
  • Never rely on a single category of stakeholder (e.g. programme managers, beneficiaries) in order to reduce strategic bias.
Top