Methodological bases
Evaluation process (How?)
Analysis

Cause-and-effect analysis

• What does this mean?
• Attribution or contribution?
  • Attribution
  • Contribution
• Analytical approaches
  • Counterfactual
  • Case studies
  • Causal statements
  • Meta-analysis
• Generalisation
• Recommendations

What does this mean?

Approach through which the evaluation team asserts the existence of a cause-and-effect link, and/or assesses the magnitude of an effect.

Attribution or contribution

Attribution analysis

Attribution analysis aims to assess the proportion of observed change which can really be attributed to the evaluated intervention. It involves building a counterfactual scenario.

Contribution analysis

Contribution analysis aims to demonstrate whether or not the evaluated intervention is one of the causes of observed change. It may also rank the evaluated intervention among the various causes explaining the observed change. Contribution analysis relies upon chains of logical arguments that are verified through a careful confirmatory analysis.
It comprises the following successive steps:

• Refining the cause-and-effect chains which connect design and implementation on the one hand, and the evaluated effect on the other. This step builds upon available explanations pertaining to the evaluated area. Explanations derive from the diagram of expected effects drawn in the first phase of the evaluation, from the evaluation team's expertise, and from exploratory analyses.
• Gathering evidence related to each link in the cause-and-effect chain, including findings of similar studies, causal statements by interviewees, and evidence from in-depth inquiries.
• Gathering evidence related to other explanations (other interventions, external factors).
• Developing a step-by-step chain of arguments asserting that the intervention has (or has not) made a contribution, and possibly ranking the intervention among other contributions.
• Submitting the reasoning to systematic criticism until it is strong enough.

Analytical approaches

Counterfactual

The approach is summarised in the diagram below:

The "policy-on" line shows the observed change, measured with an impact indicator, between the beginning of the evaluated period (baseline) and the date of the evaluation. For instance: local employment has increased, as has literacy. The impact accounts for only the share of this change that is attributable to the intervention.
The "policy-off" line, also called the counterfactual, is an estimate of what would have happened without the intervention. It can be obtained with appropriate approaches like comparison groups or modelling techniques. Impact is assessed by subtracting the policy-off estimate from the observed policy-on indicator.
The assessed impact, derived from an estimate of the counterfactual, is itself an estimate. In other words, impacts cannot be directly measured. They can simply be derived from an analysis of impact indicators.
Only a counterfactual allows for a quantitative impact estimate. When successful, this approach therefore has a high potential for learning and feedback. It is nevertheless relatively demanding in terms of data and human resources, which makes it somewhat unusual in evaluation practice in developing countries.

Case studies

Another analytical approach relies on case studies. It builds upon an in-depth inquiry into one or several real life cases selected in order to learn about the intervention as a whole. Each case study monograph describes observed changes in full detail. A good case study also describes the context in detail and all significant factors which may explain why the changes occurred or did not occur.
In a case study approach, the evaluation team analyses the whole set of collected facts and statements and checks whether they support assertions like "the change can be attributed to the intervention", "the change can be attributed to another cause", "the absence of change can be attributed to the intervention", etc.
Just one case study may fully demonstrate that the intervention does not work as intended and may provide a convincing explanation for that. However, it is worth confirming such a finding by one or two additional case studies.
On the other hand, it takes more case studies to demonstrate that the intervention works, because all alternative explanations should be carefully investigated and rejected.
If professionally implemented, case studies provide for a highly credible and conclusive contribution analysis. The approach is nevertheless fairly demanding in terms of time and skilled human resources.

Causal statements

The approach builds upon documents, interviews, questionnaires and/or focus groups. It consists in collecting stakeholders' views about causes and effects. Statements by various categories of stakeholders are then cross-checked (triangulated) until a satisfactory interpretation is reached. A panel of experts may be called to help in this process.
A particular way of implementing this approach consists in collecting beneficiaries' statements about impacts or direct results. Typically, a sample of beneficiaries is asked questions like "How many jobs would you say have been created/lost in your firm as a result of the support received?" or "To what extent is your present situation/behaviour attributable to your participation in the intervention?" In this approach the interviewee is asked to apply the policy-off scenario on his/her own.
Evaluation teams tend to prefer this approach which is far more feasible, but nobody should forget that the difficulty is transferred to the respondents. Most often, interviewees do not have a clear view of the policy-off scenario. They try to make up their minds in a few seconds during the interview and in doing so are subject to all kinds of biases.
When interviewing beneficiaries, the evaluation team often faces difficulties due to deadweight, since interviewees tend to exaggerate the effect of the evaluated intervention on their own behaviour or situation. In other words, they tend to underestimate the changes that would have occurred in the absence of the intervention. This results in a bias which is called deadweight.
In order to avoid this bias, the evaluation team should never rely upon a single naive question like "How many new jobs have been created as a result of the support received?" or "How much has your income increased as a result of the project?" By contrast, multiple triangulated questions may enable the evaluation team to assess and reduce the bias. Beneficiaries' statements are called "gross effects" (including bias) whilst the evaluation team's estimate is called a "net effect" (corrected from bias).

Meta-analysis

This approach builds upon available documents, for instance:

• Previous works pertaining to the evaluation as a whole (monitoring, audit, review, etc.)
• Recent reports related to a part of the intervention, e.g. a project, a sector, a cross-cutting issue (evaluation reports but also monitoring, audit, review, etc.)
• Lessons learnt from other interventions and which can be used in answering the question.

In performing meta-analyses, the evaluation team needs to (1) assess the quality of information provided by the reviewed documents, and (2) assess the transferability to the context of the evaluation underway.

Generalisation

The first two approaches (counterfactual and case studies) have the best potential for obtaining findings that can be generalised (see external validity), although in a different way. Findings can be said to be of general value when all major external factors are known and their role is understood. Counterfactual approaches build upon explanatory assumptions about major external factors, and strive to control such factors through statistical comparisons involving large samples. Case studies strive to control external factors through an in-depth understanding of cause-and-effect mechanisms.

Recommendation

The evaluation team should be left with the choice of its analysis strategy and analytical approach.