Counterfactual impact evaluations of Cohesion Policy
(2012)Enterprise support: support to SMEs and large enterprises in Italy, including a comparison of grants and other financial instruments
This study examined two cases of enterprise support – an investment grant available throughout Italy (Law 488) and various SME schemes in the region of Piemonte.
A particular advantage of the study was the access to several detailed administrative data sources from the Statistical Archive of Active Enterprises (ASIA) assembled by the Italian Statistical Agency (ISTAT). This, along with the availability of a clear control group in both cases (narrowly rejected enterprises for Law 488, almost random allocation between different forms of support in Piemonte) gives a high degree of confidence in the results.
In addition, a beneficiary survey was conducted to allow a comparison with more traditional methods of estimating impact.
Both monitoring data (reporting 82,000 jobs supported) and the beneficiary survey (reporting 36,000 jobs created) proved to be very poor estimators of actual jobs created under Law 488 (12,000). This adds weight to the argument to use counterfactuals (not monitoring or beneficiary surveys) to assess impact.
Impacts under Law 488 are confined to SMEs – large enterprises are using the money for projects they would have carried out anyhow. Interestingly, this is where the size effect ends – results are remarkably consistent and positive over the subgroups of micro, small and medium-sized enterprises.
Even allowing for firm size, smaller grants are much more cost-effective than larger ones: cost per jobs averaged €79,000 for the smallest grants (less than €125,000), rising to €489,000 for the largest grants (above €500,000).
The study provides a first indication of the relative merits of soft loans and grants: the loans had a cost per job around half that of grants plus a surprisingly high impact on investment – EUR 5 per euro of gross grant equivalent.
The quality of the jobs created (using productivity and payroll costs as a proxy) is usually similar to average jobs in the enterprises concerned. And, in the case of the loans, the quality is actually slightly higher than average.
(2011) Innovation support: examination of data for 9 countries, counterfactual analysis for Czech republic and Germany
This study examined publicly available beneficiary and commercial databases in various Member States. The goal was to see where data allowed a counterfactual impact evaluation of innovation support.
Of 10 cases examined, 5 were eliminated from the analysis because of small numbers of ERDF innovation projects and hence small sample size (Poland, Slovakia, Slovenia, Flanders, London and Wales). Spain was eliminated because of poor beneficiary data, France was eliminated because, although beneficiary data was otherwise complete, it was not possible to tell the year in which the support had been given.
A counterfactual analysis was therefore conducted for the Czech republic (impact on patents) and Germany (impact on R&D investment and a wide range of innovative behaviours, as measured by the Community Innovation Survey).
The study noted a range of problems encountered in the beneficiary data. It therefore made a series of specific recommendations for reporting in the future programming period, including a clear and unique firm identification as well as a description of assistance which makes clear both purpose and timing.
During the financial crisis, patent applications fell by 63% in non-supported enterprises in the Czech republic but only 14% in supported enterprises.
The subsidized firms in the German sample show a median R&D intensity of 6.2%. Without a grant from the ERDF, they would have only achieved an estimated R&D intensity of 4.4% - an increase of just over 40%.
In fact, the representative firm would have had R&D expenditure of 213,000 EUR without ERDF. The grant increases R&D expenditure to about 300,000 EUR. Thus, the treatment effect in terms of EUR amounts to 87,000 EUR, on average, for a typical grant size of up to 51,000 EUR. Although there is a margin of uncertainty here, this suggests that each euro of public money is additional and levers in more than half a euro of extra private money.
In addition, R&D grants in Germany impacted on a wide range of measures from the Community Innovation Survey, from product and process innovation to initiating new innovative projects (and not abandoning old ones).
This effect was mostly additive when considering national grants, however there was only a weak correlation with grant size (suggesting smaller grants, repeated in later years if necessary, would be particularly effective – further research would be necessary to validate this finding).
(2011) Enterprise support in Northern Ireland
This study tracked 480 firms (253 Invest NI clients and 227 non-assisted firms) for which annual GVA, turnover and data were available. The goal was to estimate the impact of assistance using counterfactuals. This was not straightforward, since supported firms can be quite different to non-supported firms (making it more difficult to find similar "matches").
However, the following tentative conclusions were reached:
There was a significant positive impact on GVA and turnover.
Impact on employment was not statistically significant, but positive at around 2% per annum. Over the period 2001-2007, employment in non-assisted manufacturing firms fell by 3.9% per annum - for assisted firms the drop was only 1.9%. There was a similar boost in the business service sector, where non-assisted firms grew by about 4.9% per annum, while assisted firms grew by 6.9%.
(2010) Enterprise and innovation support in Eastern Germany
This study compared enterprises in Eastern Germany which benefited from investment or research grants with similar, but unsupported, enterprises. There were two specific samples: the IAB Betriebspanel for enterprise support and Gefra's survey of enterprise R&D in Thuringia. To ensure robust results, various comparison methods were used (including propensity score matching, controlled difference in difference and instrumental variables). Main findings:
Investment grants induced strong investment effects. Average public support of €8,000 per employee led to €11,000-12,000 of extra investment. This implies a leverage effect, where every euro of public money generates up to €1.5 of total investment.
R&D grants of €8,000 led to an additional €8,000 of investment. Although this 1-to-1 ratio is a little smaller than that for investment grants, it has an additional "spillover" benefit in terms of increased long term regional economic growth.
A rough calculation of the direct employment effect from investment grants was some 27,000 extra jobs. While positive, this is lower than figures derived from monitoring data, suggesting that the main impact of such support is increased investment and productivity, with job creation a secondary impact.
(2010) Urban neighbourhoods in crisis – URBAN II
The second round of the URBAN Community Initiative supported 70 urban neighbourhoods in crisis, facing challenges such as high unemployment, crime, social exclusion and dereliction. The ex post evaluation included an examination of various indicators with a view to counterfactual impact evaluation. A particular focus was a simple difference-in-difference analysis of unemployment rates (a comparison of the change in the area with changes in neighbouring areas or in the parent city).
Unfortunately, the necessary area data were not available in half the neighbourhoods and the results for the small sample remaining were inconclusive. In addition, unemployment was just one target of URBAN II programmes – the tracking of other goals was often more patchy than for unemployment. This underlines the importance of programmes collecting good data on the things they are trying to change.