The result orientation: Cohesion Policy at work

You are invited to participate in the 7th European Evaluation Conference "The result orientation: Cohesion Policy at work" organized by the Directorates-General for Regional and Urban Policy and Employment, Social Affairs and Inclusion of the European Commission. The conference will highlight the benefits of evidence based policy making. It will be a showcase for the best evaluations, a place to discuss good practices in evaluation planning, and an opportunity to broaden the range of people involved in the evaluation of Cohesion Policy.

Additional tools

  • Print  
  • Decrease text  
  • Increase text  

The result orientation: Cohesion Policy at work, Sofia, BG, 16-17 June 2016.

Cohesion Policy is the EU’s main investment policy, it targets all regions and many cities in the European Union in order to support job creation, business competitiveness, economic growth, sustainable development, and improve citizens’ quality of life. The need to understand how far these objectives have been met led to a stronger focus on results in the 2014-2020 Programming Period.

The European Commission's 2016 Evaluation Conference brings together Managing Authorities, evaluation experts, the academia, and stakeholders from across Europe for discussing 2 key questions:

  • What has EU Cohesion Policy achieved in the last programming period?
  • What are the challenges to the effectiveness of the interventions in the Programming Period 2014-20?

This is the 7th edition of the conference, the first of which was held in Brussels in 1995 and the most recent in Warsaw in 2009.

At the 2016 conference evaluation evidence on the performance of Cohesion Policy will be presented, examining:

  • Lessons learned from evaluations of the 2007-2013 period;
  • The effectiveness of Cohesion Policy investments in different fields;
  • The experience of Managing Authorities with the design of Evaluation Plans and the challenges for evaluation in 2014-2020.

Agenda en pdf

Thursday, 16 June 2016

09:30

Welcoming Speakers

Address by Boyko Borissov – Bulgarian Prime Minister
Chair: Nicholas Martyn – Deputy Director General for Regional and Urban Policy, European Commission

  • Corina Crețu – Commissioner for Regional Policy, European Commission
  • Peter Pellegrini – Slovakian Deputy Prime Minister responsible for Cohesion Policy
  • Tomislav Donchev – Bulgarian Deputy Prime Minister responsible for Cohesion Policy

10:15

EU Budget and Results: the Challenges Ahead

Chair: Nicholas Martyn – Deputy Director General for Regional and Urban Policy, European Commission
Nadia Calviño – Director General for Budget, European Commission en pdf

11:15

The Role of Evaluation for the Future of EU Cohesion Policy

Chair: Francisco Merchan Cantos – Director for Audit and Evaluation, Directorate General for Employment, Social Affairs and Inclusion, European Commission
Elliot Stern – University of Lancaster and University of Bristol (UK)

12:15

Award Ceremony

Corina Crețu – Commissioner for Regional Policy, European Commission
Francisco Merchan Cantos – Director for Audit and Evaluation, Directorate General for Employment, Social Affairs and Inclusion, European Commission

12:45

Poster Session

14:15-16:00

Parallel workshops 1: existing evidence and challenges on…

Support to enterprises

Chair: Frans Leeuw, Maastricht University (NL) 

  • Dirk Czarnitzki, KU Leuven (BE) en pdf
  • Daniele Bondonio , Università del Piemonte Orientale (IT) en pdf
  • Rafał Trzciński, Polish Evaluation Society (PL) en pdf

Discussant: Rodrigo Peduzzi, European Commission en pdf

Research and innovation

Chair: Massimiliano Mascherini, Eurofound (IE) 

  • Dominique Foray, Ecole Polytechnique Fédérale de Lausanne (CH) en pdf
  • Silvia Vignetti, CSIL (IT) en pdf
  • Tatiana Fernández Sirera, Generalitat de Catalunya (ES) en pdf

Discussant: Juliet Martinez, EFTA

Institutional capacity building

Chair: Mina Shoylekova, European Commission 

  • Costanza Pagnini, Fondazione Brodolini (IT) en pdf
  • Andrea Naldini, ISMERI (IT) en pdf
  • Stéphane Jacobzone, OECD en pdf

Discussant: Nick Thijs, EIPA (NL) en pdf

Skills

Chair: Joao Santos, European Commission 

  • Karolina Jakubowska, ICF (PL) en pdf
  • Gianluca Argentin, Catholic University Milano (IT) en pdf
  • Joachim James Calleja, CEDEFOP (GR) en pdf

Discussant: Francesca Pagnossin, European Commission

16:30-17:45

Parallel workshops 2: existing evidence and challenges on…

Employment and social inclusion

Chair: Manuela Geleng, European Commission 

  • Isabel Naylon, Metis (AT) en pdf
  • Tina Weber, ICF (DE) en pdf
  • Vilija Šemetienė, Ministry of Finance of the Republic of Lithuania (LT) en pdf

Discussant: Nicola Duell, Economix Research & Consulting (DE)

Urban and territorial development

Chair: Kai Stryczynski, European Commission

  • Paolo Veneri, OECD en pdf
  • Sérgio Barroso, CEDRU (PT) en pdf
  • Oliver Schwab, IfS Institut für Stadtforschung und Strukturpolitik (DE) en pdf

Discussant: Zsolt Szokolai, European Commission

Economic and social cohesion

Chair: Daniele Vidoni, European Commission 

  • Andrey Novakov, European Parliament
  • Maximilian von Ehrlich, University of Bern (CH) en pdf
  • Guido Pellegrini, Università La Sapienza (IT) en pdf
  • Jana Jakubů, Magistrát města Brna (CZ) en pdf

Discussant: Kyosuke Kurita, Kwansei Gakuin, (JPN) University and European Commission

Investments in large infrastructures

Chair: Lena J. Tsipouri, National and Kapodistrian University of Athens (GR) 

  • Mateu Turrò, Universitat Politècnica de Catalunya (ES) en pdf
  • Aidan Coville, World Bank (USA) en pdf
  • Inesis Kiškis, Ministry of Environment of the Republic of Lithuania (LT) en pdf
  • Alex Conway, Greater London Authority (UK) en pdf

Discussant: Patty Simou, European Commission

 

Friday, 17 June 2016

09:00

The evidence on the 2007-13 Programming Period

Chair: Rudolf Niessler – Director for Policy, Directorate for Regional and Urban Policy, European Commission

  • Mariana Hristcheva – Head of Evaluation and European Semester Unit, Directorate for Regional and Urban Policy, European Commission en pdf
  • Santiago Loranca Garcia – Head of Evaluation and Impact Assessment, Directorate General for Employment, Social Affairs and Inclusion, European Commission en pdf
  • Kai Stryczynski – Deputy Head of Evaluation and European Semester Unit, Directorate for Regional and Urban Policy, European Commission en pdf

09:45

Collecting evidence on the 2014-20 Programming Period: the way forward

Chair: Andriana Sukova Tosheva – Director for investment, Directorate General for Employment, Social Affairs and Inclusion, European Commission

  • Terry Ward – APPLICA (BE) en pdf
  • Christian Biral – Conseil régional Auvergne Rhône-Alpes (FR) en pdf
  • Dominique Foray – Ecole Polytechnique Fédérale de Lausanne (CH)

11:15

Panel: Cohesion policy for results

Chair: Mariana Hristcheva – Head of Evaluation and European Semester Unit, Directorate for Regional and Urban Policy, European Commission

Address by: Iliana Ivanova – Member of the European Court of Auditors

  • Nicholas Martyn – Deputy Director General for Regional and Urban Policy, European Commission
  • Andriana Sukova Tosheva – Director for investment, Directorate General for Employment, Social Affairs and Inclusion, European Commission
  • Iskra Mihaylova – European Parliament, Chair of the Committee on Regional Development

12:45

Closing Remarks

Andriana Sukova Tosheva - Director for investment, Directorate General for Employment, Social Affairs and Inclusion, European Commission
Nicholas Martyn – Deputy Director General for Regional and Urban Policy, European Commission

Introduction by European Commission

Ladies and Gentlemen,
We firmly believe our ability to learn from evidence is among the key determinants of success. Evaluation is key in putting in place effective and efficient policies.
However, evaluation is challenging. Its usefulness depends on its quality and the involvement of all stakeholders: decision makers, beneficiaries, consultants and academia.
This capacity of the institutions to learn might be called evaluation culture: something that is far from evident in public administration.
With the idea of promoting evaluation culture the Commission launched the first evaluation competition.
Applicants have responded with 90 submissions from 18 Member States, in two strands of finalised and planned evaluations.
The European Commission is grateful for the participation to all applicants and we look forward to the new editions of the competition.


        Francisco Merchan Cantos – Director for Audit and Evaluation, Directorate General for Employment, Social Affairs and Inclusion, European Commission

Competition Winners

The competition winners are all at ex aequo in the 2 competition categories:

  • Best Completed Evaluation (Strand A)
  • Best Evaluation Proposals (Strand B)

1. Comparative Counterfactual Impact Evaluation of Financial Instruments and Grants to SMEs in Hungary

Poster en pdf

  • Name: Attila Béres & Gergő Závecz (Equinox Consulting Ltd.)
  • Type: Strand A - Best Completed Evaluation
  • Project description: There is a long debate on how to help small and medium enterprises effectively, to become more competitive, in the framework of Cohesion Policy. The evaluation compared the effectiveness of a grant and a revolving fund, in the less developed and more developed regions of Hungary, in the 2017-2013 period, targeting SMEs. Our results suggest, that revolving funds are more effective in both types of regions, and SME assistance in general is more effective in the less developed regions.

2. Contribution of the NSRF Interventions for Social Inclusion of People Living in Deprived Urban Areas

Poster en pdf

  • Name: Cohesion and Development Agency of Portugal and CEDRU – Centro de Estudos e Desenvolvimento Regional e Urbano and AMA - Augusto Mateus & Associados - Sociedade de Consultores, Lda
  • Type: Strand A - Best Completed Evaluation
  • Project description: The evaluation investigates the contribution of the National Strategic Reference Framework (NSRF) 2007-2013 for the Social Inclusion of People Living in Deprived Urban Areas; the main factors behind the decision of this study were: (i) the strategic importance attributed by the NSRF to social inclusion; (ii) high proportion of the Portuguese population at risk of poverty or social exclusion; (Iii) the strategic priority for the Europe 2020 strategy to promote inclusion and combat poverty.

3. Are personal development plans effective instruments for labour market policy? A theory-based evaluation

Poster en pdf

  • Name: ESF Agency Flanders and Katleen De Rick (KU Leuven – HIVA)
  • Type: Strand A - Best Completed Evaluation
  • Project description: Some years ago, the Flemish department of Work and Social Economy promoted the use of personal development plans to enhance lifelong employability. Several pilot projects were launched in centres for career guidance, in enterprises, in schools etc. The evaluation revealed that the government’s expectations were far too ambitious: many of the desired effects were attained, but certainly not all. The effects also varied according to the context. In the end, the government decided to focus on career guidance, where personal development plans are still used, but in other contexts the promotion of personal development plans came to an end.

4. How can the EU Cohesion Policy help persistently under-performing regions?

Poster en pdf

  • Name: Mara Giua, Department of Economics and Rossi-Doria Centre, Roma Tre University
  • Type: Strand B - Best Evaluation Proposals
  • Project description: The evaluation will look at the controversial results achieved during the decades by the European Cohesion Policy and their dependence on many different aspects. The conditioning role of both the territorial and the policy structure will be investigated: does it exist a European methodology that is driving the Cohesion Policy final territorial impact across the different scenarios in Europe? What is it made of? How do the roles of the different components of the European ‘methodology’ change in persistently underdeveloped areas? What are the discriminant factors in these areas?

5. An impact evaluation of the EU Funds on R&D in Spanish companies in 2007-2011

Poster en pdf

  • Name: Instituto de Estudios Fiscales, España. Research group: Ignacio Moral-Arce, Milagros Paniagua
  • Type: Strand A - Best Completed Evaluation
  • Project description: The aim of the study is to analyse the impact of EU Funds for R&D in Spanish companies over the period: 2007-2011. The outcome variables are the company sales, the company size, the induced investment on R&D and the number of employees on research and development jobs. Using the difference-in-differences and the propensity score matching methodologies, the study concludes that funding projects aimed at promoting R&D in Spanish companies have a clear positive impact in most of the mentioned outcome variables.

6. Evaluating the effectiveness of the Italian math teachers’ professional development programme M@t.abel

Poster en pdf

  • Name: Italian Ministry of Education - Managing Authority PON Scuola 2007/2013 and INVALSI
  • Type: Strand A - Best Completed Evaluation
  • Project description: This evaluation is a replicated randomized controlled trial, aimed at estimating the impact on students' math achievement of a teacher professional development program called M@t.abel. The project was an icebreaker for RCTs in the Italian education system and the first of an increasing lot of experiments. The evaluation also carried out a detailed implementation analysis and provided insights on features which determine or undermine the program's effectiveness.

7. Impact of EU structural assistance on quality of life, social exclusion and poverty reduction in Lithuania

  • Name: Ministry of Finance of the Republic of Lithuania / ESTEP
  • Type: Strand A - Best Completed Evaluation
  • Project description: The Evaluation assessed the EU Structural Assistance Impact on the Quality of Life, Social Exclusion and Poverty Reduction in Lithuania. It was a participatory evaluation carried out by evaluation experts of the Lithuanian company ESTEP together with the Ministry of Finance and the active involvement of different stakeholders. In order to determine the impact of the EU structural assistance on social cohesion, the impact evaluation was carried out at macro- and micro-level using theory based and counterfactual evaluation methods.

8. Can the European Structural Funds shape migration flows?

Poster en pdf

  • Name: G. Pellegrini, DISSE Sapienza University of Rome
    O. Tarola, DISSE Sapienza University of Rome
    A, Cerqua, University of Westminster, London
    G. Ceccantoni, Doctoral School of Economics, Sapienza University of Rome
  • Type: Strand A - Best Completed Evaluation
  • Project description: This evaluation will tackle the following questions: what is the impact of regional policies on migration flows across European regions differing in size, productive efficiency and population willingness to move? What is the relationship, if any, between Structural Funds and migration? The authors define a model of labour mobility and fiscal competition with a role for Structural Funds, taking into account the public goods and services supply and the labour market conditions. The model is empirically evaluated using the regression discontinuity design.

9. Counterfactual Impact Evaluation of ESF-funded Active Labour Market Measures in Lithuania

Poster en pdf

  • Name: PPMI in cooperation with the Ministry of Social Security and Labour of the Republic of Lithuania (research team: D. Jonavičienė, E. Barcevičius, V. Indilas, A. Bytautas, D. Petroka, E. Adašiūnaitė, A. Tursa, I. Budginaitė, R. Mašidlauskaitė)
  • Type: Strand A - Best Completed Evaluation
  • Project description: The counterfactual impact evaluation of ESF-funded was used to evaluate the support for the registered unemployed. The researchers analysed administrative data covering 2004-2013 of over 900,000 unemployed persons to estimate the impact of wage subsidies and vocational training on the labour market outcomes for the participants. The evaluation was financed by the European Commission under the call for proposals 'Pilot Projects to carry out ESF related Counterfactual Impact Evaluations'.

10. Ex-post evaluation of Major Projects financed in the Research, Development and Innovation Sector in the 2007-20013 programming period

Poster en pdf

  • Name: Silvia Vignetti and Emanuela Sirtori (CSIL)
  • Type: Strand B – Best Evaluation Proposals
  • Project description: The project aims to assess the rationale for supporting major Research, Development and Innovation infrastructures in a regional development perspective, and to evaluate their contribution to socio-economic welfare. It will do so with a methodology based on project portfolio review and selected case studies with cost benefit analysis and storytelling.”.

The competition is organised in two strands:

  • Best Completed Evaluation (Strand A)
  • Best Evaluation Proposals (Strand B)

For both strands of the competition, the subject of evaluation will be the effectiveness of operations co-financed by Structural or Cohesion Funds by using at least one of the following three methods:

  • Ex-post cost benefit analysis (CBA)
  • Theory-based evaluations,
  • Counterfactual evaluation.

The authors of the best evaluations and evaluation proposals will be invited to present their contributions at the Evaluation Conference that the European Commission will organise in the first half of 2016. About 500 participants will be present, including evaluation practitioners from the EU and beyond, representatives of public authorities in the Member States responsible for managing cohesion policy programmes, and academic experts - all working in and experienced with cohesion policy. The conference aims to shape Cohesion Policy by bringing together academics and practitioners. Please find a short description of each competition below.

Best Completed Evaluation (Strand A)

This strand of the competition is for evaluations that have been completed by the final deadline for submission (31.12.2015). The evaluation can be carried out by any individual or company as long as it assesses the effectiveness of EU Cohesion Policy interventions, primarily but not exclusively in the areas of Research and Innovation, Enterprise support, ICT, Large Infrastructure, Employment, Skills and Social Inclusion, or Institutional Capacity Building. The evaluations will be assessed by a jury of international evaluation experts. Please find more information in the call for papers.

Best Evaluation Proposal (Strand B)

In this strand of the competition, we ask for proposals for the evaluation of EU Cohesion Policy interventions. One condition to participate in this competition is that the evaluation has not already started. In particular, it would be welcome if the proposals concentrate on the areas of Research and Innovation, Enterprise support, ICT, Large Infrastructure , Employment, Skills and Social Inclusion, or Institutional Capacity Building. Furthermore, we are also looking for proposals that make use of new data and creative research designs. We encourage applicants to have a look at the data available on the European Commission's websites ( e.g., EU Regional Policy : Data for Research). The already available materials will be supplemented with the new results from the Ex-post evaluation 2007-2013 as they emerge. These results could be used as well as datasets from national or regional authorities who manage Cohesion Policy programmes. The proposals will be assessed by a jury of international evaluation experts. The authors of the best proposals will then be invited to present their proposals in a poster session at the conference in 2016.
This competition is open to all individuals and the participation of young researchers and PhD students is strongly encouraged.
The prize is acknowledgement of excellence by recognised international evaluation experts. There will not be a monetary prize for the two strands. At least the five best contributions for each strand will be selected, but the jury retains the possibility of widening the number of contributions selected in the case of many excellent entries. The authors will be invited to take part in the conference in 2016 with costs for travel and accommodation paid for by the European Commission according to European Commission rules.

Participation

For Strand A: Participation in this competition is open on equal terms to all natural and legal persons coming within the scope of the Treaties. It is not possible to submit to the competitions evaluations directly funded by the European Commission.

For Strand B: Participation in this competition is open on equal terms to all individuals coming within the scope of the Treaties.

Selection Criteria

Both types of competition are assessed by the same criteria:

Quality of the Methodology

This criterion will assess the appropriateness of the methodology of the whole evaluation and the specific methodologies for certain parts of it. In the overall framing of the exercise, it is important to underline how current discussions on evaluation and public policy analysis methods could be applied to get more knowledge on the impact of EU Cohesion Policy. Creativity and the use of different methods (using at least one of Ex-post CBA, Theory-based evaluations, Counterfactual evaluation) are considered relevant elements for the quality of the work. Concerning Strand B, the use of new data, available on the websites of the European Commission or from national authorities, would be a plus.

Practical Relevance

Practitioners from all institutions are not only interested in the academic debates and how to incorporate new methods and findings into practical work, but also on the results obtained from such an evaluation (or potential evaluation). This is why the second important criterion is about the potential policy significance of the findings for future Cohesion Policy. What kinds of investments promise more effects than others? What are the hidden factors under certain conditions that could significantly affect job creation? These might be some of the questions whose answers could really make a change in future EU Cohesion Policy.

Form of Submission

Best completed evaluations (Strand A):

1) Executive summary with the following structure:

  1. Research Question
  2. Description of the object to be evaluated
  3. Methodology
  4. Data
  5. Plan of Actions

The paper should be written in English and have a length of 3000 words maximum (excluding Annex and Bibliography).

2) Full evaluation report in the original language in which it was produced.

Best evaluation proposals (Strand B) should be sent in the following structure:

Executive summary with the following structure:

  1. Research Question
  2. Description of the object to be evaluated
  3. Methodology
  4. Data
  5. Plan of Actions

The paper should have a length of 3000 words maximum (excluding Annex and Bibliography), and it should be submitted in 2 versions.

  • Version 1: in English;
  • Version 2: in the official language of the applicant.

Registation website

If you have any further questions, please do not hesitate to contact the DIRECTORATE-GENERAL FOR REGIONAL AND URBAN POLICY, unit B.2. For this purpose, please send an E-Mail to: regio-eval-conference@ec.europa.eu

Privacy statement en pdf

To get the latest information on the competition and the conference and to receive all updates on new data available on our website, please register to our RegioNetwork group