Impact evaluation centre

A counterfactual is simply what would have happened without a given intervention. A control or a comparison group serves as an estimate for this counterfactual. The difference in outcomes between treated group and control group is the estimate of impact.

It is seldom enough to simply compare the treated units before and after the intervention. The world is constantly changing and we need to know what would have happened to the treated units without the intervention.

An clear example of this comes from grants to enterprise: a given firm may have made all or part of the investment even without the support. Conversely, in a time of economic crisis, some firms may have actually declined without timely assistance. Knowing which of these counterfactual scenarios actually applies makes a crucial difference to any estimation of impact.

No. A reliable control or comparison group requires a reasonable number (the statistician's "n") of treated units and of broadly similar non-treated units, in order to mobilise the power of statistics and large numbers. This condition often holds for support to enterprises and innovation as well as training and human resource measures. It seldom holds in the case of large infrastructure projects – a more appropriate quantitative method for such projects would be ex post Cost Benefit Analysis.

The single most important requirement is good data – not just for the treated units, but for enough similar non-treated units. This means that even in appropriate fields (enterprise support, innovation, training etc) it may not be possible to perform a counterfactual analysis for lack of data.

The strength of counterfactual impact evaluation lies in quantified estimates of impacts at the micro level – "how much has changed because of cohesion policy?" and in identifying the most effective investments "what type of project generates the most impact?"
But other methods also have important roles to play. For example:

• Indicators collected by monitoring systems (eg number of firms assisted, km of road built) are an essential measure of progress on the ground
• Implementation and process evaluations identify bottlenecks and show where it is necessary to improve administrative practices and capacity
• Case studies and beneficiary surveys are crucial to understanding the mechanisms driving what works (and what doesn't work)
• Ex post cost benefit analysis estimates quantified impacts for large infrastructure projects – a field where the counterfactual method would be difficult to apply
• Macro-economic models are the only tool able to assess changes generated in the broader regional economy