16/09/2010
Research impacts
Since the early 1990s, the successive Industrial & Materials Technologies programmes (and the Transport and SMT programmes when they were under the same management) have had large numbers of completed projects evaluated "ex post" by external evaluators. Over the years, this has evolved to collecting fairly detailed information about the outcomes and the achieved level of success and impact of the projects.
For each project, an independent evaluator collects information on:
- the actual achievements of the project, not only in the sense of achieving the intended RTD results, but also of achieving the intended socio-economic impact or policy support;
- the anticipated impacts in subsequent years;
- a wide range of (objective and subjective) factors which may or may not have contributed, or still contribute, to success or failure.
On that basis, (s)he
- delivers an opinion on the project's level of success in achieving its declared objectives (in terms of the intended RTD results as well as in terms of the intended exploitation and impacts), and on the factors contributing to that level of success;
- identifies where to seek for evidence of actually achieved impact in a possible follow-up assessment several years later;
- informs / advises the Commission on cases where results are left unexploited; on cases which merit promotion as technology offers or as success stories; on issues to be retained as (positive or negative) "lessons learnt", etc.
The evaluation results are collected in a database, and:
- a statistical analysis is performed to identify the main effective success and failure factors, as well as areas of higher and lower impact within the programme;
- data are processed into documents and presentations on the achievements and impact of the programme.
The main output of this exercise is a database with information on:
- the actual achievements of the projects, not only in the sense of achieving the intended RTD results, but also of achieving the intended socio-economic impact or policy support;
- a wide range of (objective and subjective) factors which may or may not contribute to success or failure to achieve the intended results and impacts but it delivers also overall analyses which have throughout time provided valuable background information when preparing the planning and documents for new activities.
The exercise also identified success stories (in terms of impact) as well as "lessons learnt". The knowledge generated by the success/failure analysis has allowed to orient the contents as well as the funding schemes of the programme towards maximal impact, and has served as background and justification for the options taken in later RTD programmes.
- Impact evaluation of Brite-Euram - IMT (FP3 - FP4) projects
- Impact evaluation of Growth (FP5) projects
- Impact evaluation of NMP (FP6) projects
- Success stories
Impact evaluation of NMP (FP6) projects
A contract procedure was launched in October 2008 to carry-out two studies to evaluate the impact of NMP under FP6. The results of the first one, an evaluatioon at strategic level, are now available: 
(3.61MB)
"Strategic impact, no revolution." - press release
Oxford Research AS from Norway and the Austrian Institute for SME Research conducted evaluation of the strategic value and impact of thematic priority “Nano-technologies and nano-sciences, knowledge-based multifunctional materials, and new production processes and devices” (NMP) in Framework Programme 6 (FP6) in its wider European and international context.
One of the most important findings of the evaluation is that the programme strategically affected Europe’s competitive position and also influenced Member States' policies and research agendas in the area of NMP. However, it cannot be directly linked to a revolution with regard to creating substantial scientific or industrial breakthroughs although these were among the explicitly targeted objectives.
Read more
Impact evaluation of Brite-Euram - IMT (FP3 - FP4) projects
The "EVIMP" study, running from early 2001 to early 2004, looked back at approx. 1200 projects that had ended in the period 1999 - 2001 (each time approx. 1 year after project end) and at more than 700 projects that had ended in the period 1996 - 1998 (each time approx. 4 years after project end).
The report that has been published is available here and a brochure summarising the results.
Impact evaluation of Growth (FP5) projects
The "EVIMP-2" study, running from mid 2004 to early 2008, looked back at 979 projects of the "Growth" programme. The mid-term results of Evimp-2 relate to 623 projects of the entire Growth programme (except Key Action 2). A summary of the findings at that stage is available here, the "EVIMP-2" brochure is available here (4.9MB ) and the report is available here (2.8MB ).
The services responsible for Key actions 3 and 4, Generic Activity 1, and Support to Research Infrastructures discontinued their participation at the mid-term of Evimp-2. The study continued for Key Action 1 and Generic Activity 1, where in total 726 projects were assessed (370 before and 356 after the mid-term; approx. 90% of all RTD, CRAFT, and thematic network / concerted actions projects). The final report (8MB ) looks only at Key Action 1 and Generic Activity 1 (i.e. the areas corresponding with the current NMP Theme, although the fraction of the projects dealing with nanosciences and nanotechnologies was in FP5 substantially lower that is currently the case).
The data collected from 724 completed "NMP like" projects, representing a total expenditure of €1,765 million, of which €992 million were EC funding, show that:
- 61 % of the projects would not have been undertaken at all without EC funding;
- 75 % of the projects were effective in achieving their scientific-technical objectives, but only 54 % were effective in achieving also at least part of their exploitation objectives;
- for 46% of the projects, the benefits were qualified by the project consortia as worth the cost of the project or substantially more;
- 41% of the projects were qualified by the evaluators as direct successes (impact certain), 31% as conditional successes (impact depending on external factors), 17% as projects with low impact but valuable results, and 12% as failures;
- 70% of projects are expecting at least some future benefits in terms of increased sales, with 41% expecting at least some benefit from cost reduction and 34% expecting at least some benefit from reduced financial risk.
In addition, the projects have generated or are expected to generate:
- approx. 3 700 prototypes / demonstrators / pilots;
- approx. 750 new software tools;
- approx. 19 000 publications;
- approx. 2 150 doctorates;
- approx. 310 technical standards;
- approx. 420 inputs into EU legislation texts;
- approx. 1 080 patent applications and 200 registered designs or other forms of IPR protection;
- approx. 240 spin-off companies;
- approx. €950 million in cost reduction;
- approx. €7.5 billion additional sales;
- the safeguarding of approx. 41 000 jobs;
- the creation of approx. 9 400 new jobs.
Evaluation and long-term impact assessment of industrial Research under FP5 (EVIMP-2): Discussion workhop Brussels, 16/11/2009
The aim of the workshop was to present, discuss and validate the results of the "Evaluation and long-term impact assessment of industrial Research under FP5 (EVIMP-2)", bringing ideas and recommendation for the future, in terms of content of the Framework programmes and on evaluation and impact assessment methodologies. The workshop included presentations related to the EVIMP-2 report, as well as of similar national initiatives, followed by discussion. A set of recommendations, proposals and open questions was amongst the results of this seminar.
Conclusions and presentations down be downloaded here.
Presentations:
- Michel Poireau: (104Kb)
- Mathieu Capdevila: (598Kb)
- Helmut Kergel: (885Kb)
- George Vekinis: (375Kb)
- Gerd Schumacher: (684Kb)
- David Flood: (217Kb)
- Jesús Alquézar Sabadie:
- Pavel Holubar: (3.42Mb)
Conclusions: (206Kb)
Attendance List: (134Kb)
Impact evaluation of NMP (FP6) projects
A contract procedure was launched in October 2008 to carry-out two studies to evaluate the impact of NMP under FP6. The results of the first one, an evaluatioon at strategic level, are now available: (3.61MB)
"Strategic impact, no revolution."- press release"
Oxford Research AS from Norway and the Austrian Institute for SME Research conducted evaluation of the strategic value and impact of thematic priority “Nano-technologies and nano-sciences, knowledge-based multifunctional materials, and new production processes and devices” (NMP) in Framework Programme 6 (FP6) in its wider European and international context.
One of the most important findings of the evaluation is that the programme strategically affected Europe’s competitive position and also influenced Member States' policies and research agendas in the area of NMP. However, it cannot be directly linked to a revolution with regard to creating substantial scientific or industrial breakthroughs although these were among the explicitly targeted objectives.
Read more
Success stories
Most success stories elsewhere on this website have been written during the lifetime of the project presented in it, and therefore before evidence of the actual use of the expected promising results is available.
The success stories linked to from this page have been written with - apart from an interesting story to tell - also evidence in hand of significant socio-economic impacts (e.g. through employment creation, product or market development, technology transfer, etc.).
