Guidance and methodological resources

November 2020: Ad Hoc seminar on GHG methodologies 2021-2027 with EIB

Agenda

• Introduction by DG REGIO
  Introduction by the EIB - Watch the video recording
• EIB: Energy Efficiency in buildings (public buildings, dwellings, SMEs) - Watch the video recording
• EIB: Renewable energy and cogeneration - Watch the video recording
• EIB: Low Carbon Transport & Rail - Watch the video recording
• Wrap up - Watch the video recording

January 2020: Ad Hoc seminar Transport result indicators 2021-2027 with JASPERS

Agenda

• DG REGIO: 2021-2027 Common result indicators for transport
• JASPERS-EIB: Indicators - RCR55 Passenger-km
• JASPERS-EIB: Indicators - RCR56 Passenger-hours
• JASPERS-EIB: Indicators - RCR64 Cycling
• JASPERS-EIB: Indicators - RCR60 Freight on Waterways
• JASPERS-EIB: Indicators - RCR58 Rail Passenger-km
• JASPERS-EIB: Indicators - RCR101 Rail Passenger hours
• JASPERS-EIB: Indicators - RCR59 Rail Freight

The European Commission has suggested a more results – oriented approach for EU cohesion funding in its legislative proposals for 2014 - 2020.

• Guidance document on evaluation plans
• Concepts and recommendations (revised 2018)
• Additional documents
  • Result Indicator Pilot Report
  • Task force on outcome indicators – prepared by a team of academics and experts to inform the reflexions on future cohesion policy :
    • Outcome Indicators and targets – Towards a Performance oriented EU Cohesion Policy (Fabrizio Barca, Philip McCann)
    • Meeting climate change and energy objectives
    • Improving the conditions for innovation, research and development
    • Guidance for the Design of Quantitative Survey-Based Evaluation
    • Guidance for the design of qualitative case study evaluation
    • Good practices in the selection and use of outcome indicators
    • Experience of the 2007-2013 Italian Performance Reserve (PR) – measurable objective for public services provision in Mezzorgiorno
• Guide on ex-ante evaluation
• Guidance on the performance framework review and reserve in 2014-2020
  
• Guidance on evaluation of innovation
• Guidance for the Terms of Reference for Impact Evaluations
• Standardised key terminology

In any programme, the crucial questions are "what do you want to change?" and "how would you know if you have changed it?" These evaluation questions are not just bureaucratic requirements, but the essence of good programming. Impact evaluation mobilises scientific and statistical tools to follow up on these questions.

Impact evaluation in DG Regional Policy falls into two broad categories:

• The "Theory-based" impact approach, which follows each step of the intervention logic and focuses on the mechanisms leading to the observed change, is particularly appropriate for answering the questions "why?" and "how?" and "in what context?" an intervention works.
• The counterfactual impact approach , i.e. the use of control or comparison groups, is particularly useful in answering "how much?" of the change is due to the intervention and comparing the effects of different instruments (or the same instrument applied to different target groups).

The two approaches are complementary and the most useful impact evaluations draw on a mix of methods: counterfactual methods to quantitatively estimate an impact, theory-based methods to understand the underlying mechanisms and the context of an intervention thus helping to modify or generalize it to other contexts.

Conterfactual approach

• Frequently asked questions (FAQS)
• Counterfactual impact evaluations of Cohesion Policy. DG Regional Policy has been working in this field since 2008 and we have a growing body of evidence for impacts. Each evaluation has a non-technical summary.
• Going deeper - reference materials and resources. Includes training opportunities and online reference works

Theory-based approach

• Frequently asked questions (FAQS)
• Guidance on theory- based evaluation
• Going deeper - reference materials and resources.

For further information, support or training opportunities, please contact regio-eval@ec.europa.eu

Working Documents on evaluation

• Working Document No 1: Indicative Guidelines on Evaluation Methods: Ex-Ante Evaluation
• Working Document No 2: Indicative Guidelines on Evaluation Methods: Monitoring and Evaluation Indicators
• Working Document No 3: Commission Methodological Paper giving guidelines on the calculation of public or equivalent structural spending for the purpose of additionality
• Working Document No 4: Guidance on the Methodology for carrying out Cost-Benefit Analysis:
• Working Document No 5: Indicative Guidelines on Evaluation Methods : Evaluation during the Programming Period
• Working document No 7: "Reporting on core indicators for ERDF and Cohesion Fund"

Other supportive documents

• Analysis of Errors in Cohesion Policy for the years 2006-2009 - Actions taken by the Commission and the way forward
• Regions 2020: An Assessment of Future Challenges for EU Regions - November 2008
• Fostering the urban dimension: Analysis of the Operational Programmes co-financed by the European Regional Development Fund (2007-2013) - November 2008
• Innovation in the national strategic reference frameworks (working document) - October 2006
• The Smart Guide to Innovation Based Incubators (IBI) - February 2010 - 20 Case studies
  Innovative strategies and actions: Results from 15 Years of Regional Experimentation
• EVALSED: The updated guide for the evaluation of Socio-Economic Development
• Handbook on SEA (Strategic Environmental Assesment) for Cohesion Policy 2007-2013:
• Checklist Water and Waste Major projects (20/11/09)

Guidance documents 2000-2006

Working Documents:

• 1.Vademecum on the preparation of Plans and programming documents
• 2.The Ex-Ante Evaluation of the Structural Funds interventions
• 3.Indicators for Monitoring and Evaluation: An indicative methodology
• 4.Implementation of the performance reserve
• 5.Verification of additionality for Objective 1
• 7.Ex Ante Evaluation and Indicators for INTERREG (Strand A and B)
• 8.The Mid Term Evaluation of Structural Fund Interventions
• 8a.The Mid Term Evaluation of Structural Fund Interventions - INTERREG III
  
• 8b.The Mid Term Evaluation of Structural Fund Interventions - URBAN
  
• 9. The Update of the Mid-term Evaluation of the Structural Fund Interventions
  

Other Guidance Documents:

• Guide to cost-benefit analysis of investment projects
  Published by DG Regional Policy in 2002
  
• Guide to the Cohesion Fund 2000-2006
• Application of the Polluter Pays Principle
• Information society and regional development : ERDF Interventions 2000/2006
• Mainstreaming equal opportunities for women and men in Structural Fund programmes and projects.

• Vademecum for URBAN II Programmes
• Application of the “n+2” rule
  2002: (~ 35KB)
  2003 :
• Guidelines on Criteria and Modalities of Implementation of Structural Funds in Support of Electronic Communications

Guidance Documents 1994-1999

• Technical assistance
• COMMISSION WORKING DOCUMENT (February 2004)- Application of Article 151(4) of the EC Treaty: use of the Structural Funds in the field of culture during the period 1994-1999 (75KB) - (75KB) + annexe (1,5MB)

• Understanding and Monitoring the Cost-Determining Factors of Infrastructure Projects (April 1998)
  (110KB)
• Measuring Job Creation – Methods for the Evaluation of the Employment Effects of Structural Fund Interventions (January 1997)
  (scanned version 600KB)

• Guide to Cost-Benefit Analysis of Investment Projects for Cohesion Policy 2014-2020
• Guide to cost-benefit analysis of investment projects Published by DG Regional Policy in 2008