Impact evaluation centre

Understanding why and under what conditions a set of interventions produces effects is of utmost importance for policy-making. Theory-based evaluations allow to assess if a programme works, how it works and in which context at the same time. Furthermore, they provide a framework to assess the impact of complex programmes integrating different interventions and can be used for activities which do not lend themselves to counterfactual impact evaluations.

A theory-based impact evaluation focuses on programme theories, i.e. the assumptions of policy makers and stakeholders on the preconditions, mechanism and context for an intervention to work. Theory-based impact evaluations test these assumptions against the observed results following the different steps of the intervention logic and examine other influencing factors. They are thus able to explain why and how results have occurred and to appraise the contribution of the programme and of other factors.

When designing a programme, Managing Authorities are requested to explain how allocating funds will produce outputs through which intended results are intended to be achieved (the expected change). This theory of change has to take on board the economic and political context of the programme as well as other factors (social, cultural, institutional…) that may influence the mechanisms leading to the results.

First, it is important to check to what extent the programme contributes to the result. Positive results may be caused by a change in the context or other policies or programmes. Contrary to counterfactual methods which infer causalities from quantified comparisons between (for example) a treated and non-treated group , theory-based approaches test each assumed causal link between the intervention and the observed result through different methods (such us literature review, surveys, interviews, case studies). This also includes discarding rival hypotheses.

Second, theory-based evaluations analyse the context of the intervention and the mechanisms leading to the results. A successful intervention in a specific context (an area, a group of stakeholders, a period of time) may trigger different mechanisms in another context (because of differences in actual or perceived needs, in behaviours, in institutional settings etc.). Policy-makers willing to replicate an intervention or to improve a policy need to understand the context and the underlying mechanisms explaining the results, be they positive or negative, expected or unintended.

Yes. However it may prove difficult where the programme is not built on explicit assumptions and does not articulate a clear theory of change. In this case, evaluators need to spend part of their time (and budget) in reconstructing the theory and designing plausible result indicators guiding the evaluation.

Observational methods and fieldwork (often used in theory-based evaluations) are time-consuming. For this reason, theory-based approach should not attempt to test all the mechanisms of change triggered by a programme. Evaluation should focus on mechanisms which are not backed by previous research (e.g. in the case of new or complex interventions). What should guide the choice of the evaluation questions and methods are the needs for policy learning.

Theory-based impact evaluations do not focus on estimating quantitatively how much of the result is due to the intervention like counterfactual evaluations. Through different methods (such as "contribution analysis" or "general elimination methodology") they assess the plausible contribution of an intervention to observed changes. Theory-based evaluations can incorporate counterfactual tools to test whether the result can be attributed to the intervention but will also use observational and analytical methods to explain the mechanisms leading to this result and the influence of the context.

The strength of theory-based evaluations is that they can offer a framework to guide the use of different tools. For example:

• Indicators collected by monitoring systems (e.g. number of new jobs created in supported entities, increase in number of regional patents) which are an essential measure of progress on the ground
• Counterfactual methods to estimate quantitatively the difference made by the intervention: they are essential for evaluations aiming to establish the "value for money" of a policy
• Social network analysis which map the collaborations between different actors at different points in time thus showing an evolution
• Ex post cost benefit analysis which estimates quantified impacts for large infrastructure projects

Macro-economic models are based on assumptions on how a policy brings changes in the broader regional economy. In this sense they are theory-based tools. A first difference with theory-based evaluations is that these assumptions are those of the modelers (based on scientific researches), not of programmers and stakeholders. A second difference is that models take these assumptions as given and do not aim to test them. On the contrary, theory-based approaches allow for learning from the field and adapt theories to surprises in observed results.

The debate between quantitative versus qualitative methods is a false debate. The most useful impact evaluations use a mix of methods. Qualitative fieldwork is needed to guide quantitative work while quantitative techniques can prepare the ground for –or challenge- qualitative work. However theory-based evaluation core tools aim to explain how and why observed results occurred, be they intended or unexpected (e.g. literature reviews, interviews, surveys, ethnographic investigations, focus groups, case studies…).