IMPORTANT LEGAL NOTICE - The information on this site is subject todisclaimercopyright notice
Evaluation
  EUROPA > European Commission > EuropeAid > Evaluation > Methods > How
Last updated: 09/11/2005
EuropeAid

Methodological bases
Evaluation process (how?)
Methodological design

EuropeAid
 

Feasibility (evaluability) of a question

 


Methodology
• Evaluation Guidelines
• Methodological bases
• Evaluation tools
• Examples
• Glossary
• Sitemap

Methods
• What
• When
• Why
• Who
• How

How
• Overview
• Strategy
• Questions
• References
• Design
• Data collection
• Analysis
• Judgment
• Quality
assurance

Design
• Overview
• Design tables
• Overall design
• Developing tools
• Feasibility

 
 

What is this?

Certain questions are easy to answer while others may raise evaluability problems. It is therefore necessary to assess the feasibility of evaluation questions from the outset.

What is the purpose?

  • To ensure that the evaluation provides reliable and credible answers to the questions asked.
  • To exclude or amend questions that are too difficult to answer.
  • To adjust the available time and other resources in case difficult questions have to be answered.

What has to be taken into account?

To establish whether a question is evaluable, we check:

  • Whether the concepts are stable (Are the main terms of the question understood by everyone in the same way?)
  • Whether explanatory assumptions can be identified (What are the external factors and the cause-and-effect assumptions?)
  • Whether access to the field and data collection entail major difficulties.

Top

What are the most frequent limitations?

A highly innovative activity

If the question concerns an innovative instrument, activity or procedure, the following difficulties may arise:

  • It is difficult to define the terms of the question without ambiguity.
  • There is a lack of expertise to understand the cause-and-effect mechanisms.

A very recent activity

If the question concerns a recently implemented activity:

  • The activity has not yet produced observable effects
  • The informants have not yet stabilised their opinions.

Managerial weaknesses

If the question concerns activities in which there are or were managerial weaknesses, the following difficulties may be encountered:

  • The monitoring data and reports are inadequate or unreliable.
  • The managerial difficulties have generated conflicts that limit access to certain informants or cause those informants to bias their answers.

In case of a strong suspicion of illicit or illegal practices, it is preferable to postpone the evaluation question for later and to start with an audit.

A scope that is too complex

If the question concerns a multi-dimensional or multi-sectoral scope, the following difficulties may be encountered:

  • In view of the available time and budget, there are too many data to collect, informants to meet and analyses to perform, and they are too dispersed.

A far-reaching impact

If the question concerns a far-reaching impact which is connected to the evaluated activity through a long chain of causes and effects, then the following difficulties may be encountered:

  • There are so many external factors and they are so influential that it becomes impossible to analyse the contribution of the intervention.

An intervention that is too marginal

If the question concerns a very small activity compared to other neighbouring policies or to the context, the following difficulties may be encountered:

  • " The evaluated activity does not attain the critical mass that would allow an analysis of its contribution.

Top

Recommendations

A question is rarely unevaluable in the absolute. It is more likely to be an accumulation of difficulties and constraints that makes the question difficult.
When a question is considered too difficult, it is often possible to amend it to make it more evaluable. This can be done, for example, by limiting the scope of the question or choosing to apply it to a less distant effect or to a probable effect if the real effect is not yet observable. This option may be preferable to exclusion of the question.
Policy-makers or any other actor may insist on asking a question that is clearly too difficult. In such an instance, it is useful to provide a full explanation of the difficulties and to show that evaluations have limitations and cannot answer all questions.

Top