EASME

List of SME Instrument evaluators published!

List of SME Instrument evaluators published!

Printer-friendly versionPDF version
07/09/2016

The list of evaluators who select the most promising projects for the SME Instrument has been published today. The list contains more than 1200 experts that have worked for us in 2015. Evaluators are categorised by nationality and topic.

Publishing the list is meant to increase transparency and avoid conflicts of interest. This is of course part of a set of measures taken to ensure funds are distributed in an equal way. So how does the system work? 

High-level expertise

Submitted proposals are evaluated by independent high-level experts in the domains corresponding to SME Instrument topics. The more proposals there are for a certain topic, the more experts in that field we need. This is why more than 20% of evaluators are experts in ICT – the most popular topic under the SME Instrument.

The vast majority (above 80%) of our evaluators come from the private sector and almost half are women. In total almost 60 different nationalities are represented with around 10% of evaluators from non-EU countries.

Experts can apply to be evaluators through a call for expression of interest.

Each project submitted to the SME Instrument is evaluated remotely by four independent experts offering a mixed profile in innovation, business and finance in the topic concerned. Each evaluator works independently and there are no contacts between the four evaluators.

Projects are evaluated against the three award criteria: impact, excellence and quality of implementation. The experts-evaluators score 21 different questions and the score received by a project is the 'median' of the four individual scores. To be selected for funding, the project must score above the threshold (13 for Phase 1 and 12 for Phase 2) and be ranked among the best projects. This is all detailed in the rules of participation to Horizon 2020.

Independence and impartiality

To avoid the risk of conflicts of interest, experts submit to a code of conduct that asserts they work in an independent, impartial and confidential way. Experts must confirm there are no conflicts of interest for the work they are carrying out and they have to declare any conflicts of interest arising during their work. The Commission then assesses whether a conflict of interest exists and if so replaces the expert. If it becomes clear that a conflict of interest occurred and was concealed by an expert the work carried out is invalidated and sanctions will apply. The rules can be found in the contracts signed by experts.

Consistent evaluations

Resubmissions of proposals give us a reliable indication that the evaluation system is consistent and the candidates are treated in an impartial way. Under the SME Instrument, applicants can resubmit their application as many times as they wish. These resubmitted applications are evaluated by experts different from the first time. If the scores of resubmitted projects would differ drastically from the previous ones, especially in cases where only minor changes have been made, the validity of these evaluations could certainly be put to question. But analysis shows that the majority of scores do not change substantially. There is also a yearly rotation of 20% of the experts to ensure there's an impartial treatment of the projects submitted.

More information:

Read more and find all the relevant documents on the participant portal pages on experts.

Comments

  • Jose's picture

    The information provided here does not state which topic they evaluated, therefore making it hard to really identify any possible conflict of interest. However, the only winner of a croatian SME instrument just happens to have their CEO as an evaluator.

  • Bernd Reichert's picture

    The topics are not listed because this doesn't make sense. Evaluators work on competences not topics. And certain competences can be used in various topics, like business developments, finance, etc. Each proposal is evaluated by 4 experts who are not of the same nationality as the proposing company. As we are applying medians, you need to convince at least 3 experts of the 4 to recommend the application for funding. Two experts are technology oriented and the other two are finance/market experts.

  • Julia's picture

    I agree with Jose's comment: the topics are not listed.

    I also see in there mainly consulting companies and hardly any venture funds and well-known entrepreneurs.

    Besides, many are listed as 'European Commission' and as organisations acting as NCPs e.g. Invest Northern Ireland, Netherlands Enterprise Agency, Vinnova, Scottish Enterprise, etc. I thought all of these were very clear conflicts of interest??

    There are also many ministries. Are they considered best place to spot the next Skype?

    We'd be grateful if you could address our concerns here in the open and not dodge the difficult questions.

  • Bernd Reichert's picture

    We have currently (status 2016) 50 active investors (with traceable deal flows) in the pool. We are interested to increase this number and to expand the pool of experts in that respect, which is not a straightforward task. The published list contains everybody in the pool in 2015. On the participation of NCPs/ministries as experts, those persons received a contract in early 2015 but were excluded from the evaluation as soon as we discovered a potential conflict of interest. The SMEI pool was made of 1200 experts where around 950 experts were regularly used in 2015. This means that some experts did not evaluate any proposals in case they were found in a potential situation of conflict of interest and/or did not provide the expected quality of work. The purpose of the publication of such lists is exactly to improve the pool over time. However, it should be noted that under the existing regulation, an expert is used based on her personal competences not institutional ones. So, unless the person is an NCP in persona, there is not formal CoI. However, as said above we try to minimize conflicts without jeopardizing the scheme (you need to start somewhere).

  • Julia's picture

    I agree with Jose's comment: the topics are not specified.

    I see mostly consultancies in this list and hardly any venture firms or well-known entrepreneurs which was supposed to be our target population.

    Besides, many are listed as 'European Commission' or as organisations acting as NCPs e.g. Invest Northern Ireland, Netherlands Enterprise Agency, Vinnova, Scottish Enterprise, etc. which is a listed a CLEAR conflict of interest in your expert contract above.

    We would be grateful if you could address our serious concerns for fairness here in the open.

    Could you please also stop deleting this comment because it makes you uncomfortable?

  • martizr's picture

    We have not deleted any comment on this page and by the way your comment appears two times.

    The head of the SME Instrument unit will reply on this page in the coming days.

    EASME Communication team

  • maria's picture

    I would like to ask you for clarification about the quality of work ( mentioned in the previous message) provided by the evaluators and if , in the event of poor quality of work which may lead to the exclusion from the pool of expert evaluators, the evaluators receive a formal notification.

    Thank you.

  • storejo's picture

    Dear Maria The selection of the experts depends according to their availability, the number of working days indicated in their contract, the possible conflict of interest notified or not and their profile. The experts are briefed on what is expected from him/her notably on the quality of the evaluation.
    We have put in place a reporting system of the evaluations done by the experts. If an inappropriate behaviour is spotted, such as standard short comments ("yes", "good", etc.) or just repeating the text of the description of the sub criteria, the expert is contacted to draw his attention of this. A follow-up is done at the next cut-offs.
    If a non-respect of the contract is detected the contract can be legally terminated and therefore the expert is notified.

  • Julia's picture

    Thank you for your answers. I conclude from them that:
    1/ the large majority of funding decisions is made by people who have *zero* investment experience.
    2/ there must be something in your system that puts off investors. Any investor with normal business sense would jump at this opportunity to get deal flow. The fact that you are finding it difficult to recruit them is a strong hint that something is not working and you should really look into this.
    3/ you have no *qualitative* assessment of evaluators. You allocate billions of taxpayers' money (our money!) based on experts recommendations without actually checking how good their judgement is. Do you realise how insane this sounds?!

    I would welcome a transparent explanation from Bernd Reichert on who selects those experts and how they are evaluated.

  • Maurizio's picture

    Hello

    Could you answer to Julia's questions? Her questions concern us all and are much more relevant to us than you recent communication survey.

    She is asking important questions and all of us here are interested in what you will answer.

    Thank you

  • maria's picture

    Dear Sir/Madam,

    I think it is extremely important to provide clarification about the issue raised by Julia. I am waiting for an answer from your side.

    Thank you.

    Maria

  • martizr's picture

    We invite your to continue this conversation on LinkedIn. Mr Reichert will reply to your questions and comments there: https://www.linkedin.com/pulse/we-have-published-list-sme-instrument-eva...

    Erwan @EASME

  • Julia's picture

    I'm afraid it is not in our interest to continue the discussion on LinkedIn. From the little we see, your system is far from fair and transparent and we do not want our company to be black-listed for EU funding, just because we dared challenge the system.

    We would like that Bernd Reichert answers here.

  • maria's picture

    The selection of experts is a crucial phase in the evaluation of proposals and this web page is the appropriate channel to give us clarification.
    I am waiting for your answers.

    Thank you.

  • mark's picture

    it is not obvious to me if the list of evaluators is the same as those for 2016. we need to specify in form part a whether we have objections against evaluators/reviewer - but how do identify the pool of reviewers. is it the same as 2015? if not, where to find the list?
    regards,
    mark

  • storejo's picture

    Dear Mark, Applicants can exclude up to 3 persons and/or organisations from the evaluation of their proposal on the basis that the persons and/or organisations to be excluded are closely related or in competition to their activities. There is no direct link between the 2015 list of experts and the persons/organisations to be excluded from the evaluation of proposals. I hope this helps, Johanna@EASME

  • Katy's picture

    I have noticed that external Slovenian project evaluator under Horizon 2020, SME instrument write proposals for SME Instrument Phase 2 for SME and consult to SME how to write project to be funded. And he receives payment for this work. We nicely ask you if this is allowed him to do? We are concern that this is a conflict of interest as he is project evaluator under SME instrument.
    I would be grateful for your feedback. Best regards.

  • faturma's picture

    Dear Katy, thank you for flagging this up. We have identified this problem and as a result the cooperation with the evaluator has been terminated.
    Marie @ESME

  • Mark's picture

    Dear Marie. I am also an evaluator and sometimes consult SMEs who are applying for the SME-Instrument. If I don't evaluate any projects in that call - why would this be a conflict of interest? What are the grounds for you to terminate the contract of an evaluator if there is no conflict of interest in the evaluations we do?

  • faturma's picture

    Dear Mark, if you are not involved in the particular cut off, you are not in conflict of interest. Best, Marie @EASME

  • Daniel's picture

    But you already eliminated an external Slovenian project evaluator under Horizon 2020, SME instrument as you declared on previous comment.

  • Obvious McObvious's picture

    There is a clear conflict of interest if the people who are supposed to be the referees (the evaluators) are also coaching the teams. It doesn't matter which game you referee or what your capacity is as a coach. Evaluators decide where the money is spent. The idea that they can advertise "pay me and I will help get your proposal funded" is a disgrace, and that the Commission pretend it is not a problem, because it is impossible to work out who evaluated what proposal or when, and therefore whether a conflict of interest has taken place, is short sighted in the extreme. The Croatian example is easy to understand. The only company to be funded under the SME instrument has an evaluator as their CEO. As evaluators are selected by their relevance for the projects, it can be assumed that the CEO was able to read several applications with similar technology, or in the same sector, before submitting their bid, therefore getting an inside track on the best way to present a proposal, example texts of what to include in each section, as well as detailled briefings from the commission on how to evaluate. Did the commission investigate whether this evaluator continued to evaluate while their project was being evaluated?

    This list provides the image of transparency, but is far from transparent. It should include the topics and call dates that the evaluators evaluated for, which would make it possible to identify externally whether any conflict of interest may have resulted in a project being funded (there still may be COIs but not resulting in funding). This is essential as there is no evidence that the commission is doing anything to investigate this themselves. If they are, it would be welcome that they published information on the number of COIs identified.

  • Katy's picture

    Thank you for your quick respond and action. There are also other Slovenian evaluators which are doing the same thing. Best regards.

  • Patrick's picture

    After a first submission of a Phase 2 application we got a score of 13,6 (well above the threshold, but of course no financial support...). After some minor modifications to improve one or two weaker points, the new score after resubmission was even below the threshold... Where is the consistency in that? Very, very discouraging...

  • storejo's picture

    Dear Patrick,
    Sorry to hear your project didn't get funded. The applications are evaluated each time by different evaluators, so the scores can vary from ne cut-off to the other. Most of the time the variations are mild and extreme changes in scores are rare. For more information you can have a look at the Report on the two first years of implementation of the SME Instrument. https://ec.europa.eu/easme/en/news/sme-instrument-500-million-invested-m...
    Best regards Johanna@EASME

  • elena's picture

    Dear Sirs/manames,
    we are preparing a project proposal to be submitted under a H2020 call (innosup 01) and are wondering if we can use the list of external evaluators as a source to select the innovation coaches needed in our activities (instead of creating a new one).
    I'm waiting foreward to your kind reply,
    Elena

  • martizr's picture

    Hello,
    This information is public so anyone can use it. However, it's probably better to provide a list of real coaches whose profile has been chosen for the purpose at hand.
    Erwan @EASME

  • Paul's picture

    Hello

    Could you please share the list of experts for 2016 and 2017? The 2015 list is not very helpful for a 2017 submission...

    Thank you

    Paul

  • storejo's picture

    Dear Paul, the list of experts for 2016 will be published in summer 2017. Best regards, Johanna

  • Andreas's picture

    The SME instrument concept is great, though its implementation is rather inferior:
    - Evaluation results are random, because there are no clear guidelines, the independent experts are interpreting the requirements differently from one another. A consensus meeting, like for most other instruments would help, but here it is completely missing.
    - Proposal guidelines are almost misleading if compared with the ESR. Evaluation results (the content of the ESR) are useless, do not provide any useful information for the applicants, unlike in most other programs.
    - Experts are kind of amateurs, and unlike in case of professional VC funds cannot be made accountable for their decisions. Many of them are in for the money and not for the success of the program. Quality control of the process, at least a publicly known one, is missing. The way the program is operated is expensive, inefficient and unreliable.
    - As scoring is completely inconsistent and unpredictable, it is no wonder that many professional EU project consultants are not willing to provide advice for preparing SME instrument proposals.

    The program needs a complete overhaul and a professional operation to use the funds more efficiently, and to provide a better service for the SMEs.

    Ps.: Please show the last comments on the top of the list and not at the bottom.

  • Mark's picture

    we have conducted very professional cometitors analysis supported by the final report prepared by Deloitte & Touche.
    After evaluation process we suprisingly noticed that we are belowe threshold.
    Could you please explain how can we validate why it was assessed thhhis way and by whom ?

  • storejo's picture

    Dear Mark, please contact EASME-SME-HELDESK@ec.europa.eu about your specific case. Best Johanna@EASME

  • Teresa's picture

    hello, when are you going to update the list of evaluators? it is obsolete. Thanks

  • storejo's picture

    Dear Teresa,
    The list of evaluators for 2016 will be published in summer 2017. Best regards, Johanna@EASME

  • Leslie's picture

    Hello

    Could you be more specific on when the list of 2016 evaluators will be published in 'summer 2017'? Summer is starting in 2 weeks :)

    It seems that it is the list of 2017 evaluators that should be published in 2017. Why all this delay and secrecy?

    Thank you for your answer,

    Leslie

  • storejo's picture

    Dear Leslie,

    Thank you for your question. We expect that the list of 2016 evaluators to be published in the Participant Portal in June 2017. For transparency reasons we have a legal requirement to publish the information in the year following the evaluators' participation. I hope this helps. Best Johanna@EASME

  • Andreas's picture

    It seems that you failed to respond to my comments of the 20/03/2017 10:42. It was propably an oversight from your side, as you replied to all other comments. Can I have an answer please?
    Thanks
    Andreas

  • storejo's picture

    Dear Andreas,
    Many thanks for your comments. We don't usually comment on statements, we try to reply when people have concrete questions. As regards changes to the programme, the next Horizon 2020 Work Programme will bring some changes to the SME Instrument. We will communicate about them in detail once the Work Programme has been adopted. For an overview you can have a look here: http://ec.europa.eu/research/index.cfm?&na=na-280217&pg=newsalert&year=2017
    Best Johanna@EASME

  • Leslie's picture

    Many thanks Johanna!

  • Leslie's picture

    Dear Johanna
    Today is July, 1 and the list of evaluators 2016 still hasn't been published although you said it would be published in June :(
    Could you confirm that it will be done in the next days?
    Many thanks!
    Leslie

  • storejo's picture

    Dear Leslie, apologies for the long wait, the list will be published in the days to come. Johanna@easme

  • Leslie's picture

    Thank you for the follow-up, Johanna.
    Leslie

  • Leslie's picture

    Hi Johanna

    Could you give us an update on when the 2016 list of evaluators will be published? Why is there such a delay?

    Thank you in advance.

    Leslie

  • storejo's picture

    Dear Leslie,
    The list is still being compiled. It will most probably still be published in July 2017.
    Best regards,
    Johanna@EASME

  • Leslie's picture

    Dear Johanna
    It is now 1 August and still no list has been published...
    You are very demanding towards us applicants to the programme so it would be great if you too could stick to your commitments.
    Thank you in advance to let us know when you will deliver.
    Regards,
    Leslie

  • storejo's picture

    Dear Leslie,
    Drawing up and publishing the list of evaluators is a Horizon 2020-wide exercise that involves several services of the European Commission and various Agencies. Therefore the publication is not entirely under our control. As you may know, according to the rules of participation of Horizon 2020, the list of expert evaluators shall be published at least once a year but there is no commitment on the publication date. We are trying to give you the best possible indication of when the list will be published but unfortunately there are delays that don't entirely depend on us.
    Many thanks for your patience.
    Best regards Johanna@EASME

  • Teresa Alamos's picture

    Dear Johanna, any news about the updated list?

  • storejo's picture

    Dear all,
    There has been exceptional delays regarding the list. It will be published soon but unfortunately we cannot give you a more precise date. Best Johanna@EASME

  • Leslie's picture

    Johanna

    The word 'exceptional' is not too strong to qualify the delays we are experiencing in receiving a list of evaluators for *2016* (not even this year).

    How could we not interpret this as the Commission realising retrospectively that it should never have selected some of its evaluators ? What other reason could there be for your obvious discomfort in publishing this data ? This is quite worrying for us.

    We count on you to deliver on your promise and publish this data ASAP.

    Thank you

    Leslie

  • storejo's picture

    Dear Leslie,
    According to the rules of participation of Horizon 2020, the list of expert evaluators - of the previous year - shall be published at least once a year. Hence we will publish the list of experts used in 2016 in 2017 and the list of experts used in 2017 in 2018. As previously explained there is no commitment on the publication date. We are striving to publish the list as soon as possible however due to exceptional delays we have not been able to do so yet. Thank you for your understanding. Best regards, Johanna@EASME

  • Teresa's picture

    Dear Johanna, Just one question, if you publish the applicable evaluators list a year after the calls, it is impossible for any applicant to know which one might have a conflict of interest with your proposal. In part A you ask for indicating the evaluators that you might want to forbid, for instance evaluators related with your competitors and so on, but it is not way to know with this system if something like that might happens. Cheers.

  • Leslie's picture

    Hello Johanna

    Is there any hope that the 2016 list of evaluators will be published before 2018? As pointed out above by Teresa, such delays tend to defeat the purpose of publishing the list in the first place... You are making it really difficult for us to trust the system you have built.

    Thank you for your consideration,
    Leslie

  • storejo's picture

    Dear Leslie, We are following up on this, together with the other services concerned. The list should be published soon but we are unable to tell you when exactly. Many thanks for your patience. Johanna@EASME

  • Xavier Aubry's picture

    Indeed, publishing the list of 2016 evaluators before the EIC starts, and making sure no proposal writer or reviewer is also an evaluator in 2018 is critical.
    SME instrument consultants or reviewers simply CANNOT be SME instrument evaluators, regardless of topics and timeline. Even if you helped write a proposal 18 months ago, that proposal might be in resubmission now, while you are still in the evaluation pool, which is still by definition a conflict of interest. In addition, there are no more topics next year so all proposals will be in competition with each other. So, it’s time to choose your side: evaluator or consultant.
    For more on this: https://www.linkedin.com/pulse/h2020-sme-instrument-evaluations-good-bad...

  • Leslie's picture

    Hello Johanna and hello Mr Aubry

    I find Mr Aubry's post above of the utmost hypocrisy as I have a previous version of your database of evaluators from 2015 where his name features! Is the European Commission complicit in this whole thing?!

    This means that many of the successes that you brag about, Mr Aubry, were achieved while you were yourself an evaluator in 2014-15, a situation which you denounce in your LinkedIn post and which the European Commission seems to endorse!

    This sound very much like the Paradise papers unfolding before our eyes: do as I say but not as I do...

    Maybe your biggest actual concern here, Mr Aubry, is to make sure that competing consultants do not benefit from understanding the system as you have through being an evaluator at the beginning. Instead of throwing mud at other people, you might want to examine your own behaviour first and check how clean your own conscience really is.

    I also have a serious problem with the European Commission endorsing this kind of behaviour, erasing compromising names from a list, 'picking their consultants' by proposing an accreditation system, without having much clue about what actually happens behind the scenes. I think applicants are probably smart enough to decide by themselves if they are dealing with decent consultants or with conmen.

    Clearly, a serious rethink is needed here but from experience, people depicting themselves as morally superior are often the worst of the lot.

    Leslie

  • Jesus Rey's picture

    Dear Johanna
    Could you please update us on when the 2016 list of experts will be published? We have been waiting for months now.
    Why don't you publish the 2017 list at the same time since we are nearly at the end of the year? That would be even more helpful.
    Thank you in advance for giving us clarity on this important question.
    Jesus Rey

  • storejo's picture

    Dear Jesus, Apologies there has been delays with the list. The publication doesn't depend on us but the other services involved. The lists of remote evaluators are always published the following year, meaning that the 2017 list will be published sometime mid-2018. However, we will publish the list of jury members for Phase 2 Step 2 interviews in early 2018. Best regards, Johanna